DOCUMENT RESUME 



ED 391 590 



PS 023 819 



TITLE 



INSTITUTION 



SPONS AGENCY 

PUB DATE 
NOTE 
PUB TYPE 



National Roundtable on Outcome Measures in Child 
Welfare Services (2nd, San Antonio, Texas, April 7-9, 
1994): Summary of the Proceedings* 

American Humane Association, Englewood, Colo. 
Children's Div. ; American Public Welfare Association, 
Washington, DC* National Association of Public Child 
Welfare Administrators* 

Texas State Dept* of Protective and Regulatory 
Services , Austin* 

95 

129p* 

Collected Works - Conference Proceedings (021) 



EDRS PRICE 
DESCRIPTORS 

IDENTIFIERS 



MF01/PC06 Plus Postage* 

’'Child Advocacy; Children; *Child Welfare; *Family 
Programs; Program Effectiveness; *Welfare Services 
American Humane Association; Family Preservation 
Services; National Association Public Child Welfare 
Admin; ^Program Objectives 



ABSTRACT 

The second annual roundtable represents an on-going 
effort to share knowledge and information and to develop a common 
agenda and measure the achievement outcomes in child welfare* This 
roundtable's goals included: (1) reviewing and selecting appropriate 
outcomes for children and families in child welfare systems; (2) 
building understanding of a conceptual framework for identifying key 
outcomes; and (3) building agency and community consensus regarding 
outcomes* Following acknowledgements and an introduction, this 
summary of proceedings is comprised of the following sections: (1) 
"Roundtable Staff"; (2) "Opening Remarks"; (3) "Presentations of 
Experience," including presentations from Colorado, Texas, 

California, and Utah; (4) "Summary of Work Group Sessions," focusing 
on child safety, family continuity, preservation, child functioning, 
and family functioning; (5) "Getting Started" on some measurement 
issues ^o consider when developing outcome measures for child welfare 
services; and (6) "Summary of the Roundtable*" (BGC) 



it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it "it it it it it it it it it it it it it it it it it it it it it it it it it it it 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document* 

it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it ii it it it it it it it it it it it it it it it it it it it it it it it it it 



ED 391 590 



ECOND NATIONAL ROUNDTABLE 
ON OUTCOME MEASURES 
IN CHILD WELFARE SERVICES 




IftiMARY OF THE PROCEEDINGS 



Presented by 
American Humane Association 

AND 

National Association of 
.^BL ic Child Welfare Administrators 

- AsArnuAnof 

AmUtCAN PUBUC WUFARE ASSOCIATION 



San Antonio, Texas 
April 7 - 9 , 1994 

Co-sponsored by 
THE Texas Department or 
'£ AND Regulatory Services 



• PERMISSION TO REPRODUCE THIS 

material has been granted by 






TO THE educational RESOURCES 
INFORMATION CENTER (ERIC) ' 



2 



American 
■« Humane 
Association 

Children s Division 



mrsT ropy AVAILABLE 





V 



SECOND NATIONAL ROUNDTABLE 
ON OUTCOME MEASURES 
IN CHILD WELFARE SERVICES 



Summary of the Proceedings 



Presented By 
American Humane Association 

AND 

National Association of 
Public Child Welfare Administrators 

An ArmuATE or the American Public Welfare Asscoation 



Co-sponsored BY 
THE Texas Department of 
Protective and Regulatory Services 




3 



The views and opinions that are presented in the enciosed papers and articies do not 
necessarily reflect those of the American Humane Association (AHA), the National 
Association of Public Child Welfare Administrators (NAPCWA) or the Texas 
Department of Protective and Regulatory Services. 



® 1995, American Humane Association 



TABLE OF CONTENTS 



• ACKNOWLEDGEMENTS 

• INTRODUCTION 

• ROUNDTABLE STAFF 

• OPENING REMARKS 

Welcome and Opening Remarks 

Kittie Arnold, Arapahoe County Department of Social Services, 

Littleton, Colorado 

A Framework for Outcome Measures in Child Welfare Services/Briefing Paper 

Nancy McDaniel, Children's Division, American Humane Association 
Dennis Orthner, Human Services Research and Design Laboratory, 

University of North Carolina at Chape! Hill 

• PRESENTATIONS OF EXPERIENCE 
COLORADO 

-- Colorado Child Welfare Outcome Measures Workgroup 

Background and Accomplishments — Ann Kelley Anderson, state of Colorado Department 

of Social Services 

Three Desired Outcomes -- David Bernstein, Denver County Department of Social Services 
The Colorado Model — Terri Bailey, The Piton Foundation 

A Look at Indicators — Kittie Arnold, Arapahoe County Department of Social Services, 

Littleton, Colorado 

Implementation Issues -- Lloyd Malone, EI Paso County Department of Social 

Services, Colorado Springs 



TEXAS 

-- Operationalizing an Outcome Management Process Holistically Or, How to Add 
O.O.M.P.H. to Your Organization 

David Sheets and Homer Kern, Texas Department of Protective and Regulatory Services 



O 



5 



CALIFORNIA 



-- Effects of Numbers of In-Person Visits on Rates of Child Maltreatment Recurrence 
Will Johnson, Alameda County Social Services, Oakland 



UTAH 

" Utah Customer Survey 

Earl Bassett, Utah Division of Family Services 

" Recommended Outcomes for Residential Care 
Eric W. Bjorkiund, Utah Youth Village 



• SUMMARY OF WORK GROUP SESSIONS 

• Work Group Draft Indicators 

-- Child Safety 

- Family Continuity/Preservation 
-- Child Functioning 

- Family Functioning 

• GETTING STARTED 

• Some Measurements to Consider When Developing Outcome Measures for Child 

Welfare Services 

Raymond S, Kirk, Human Services Research and Design Laboratory 
University of North Carolina at Chape! Hill 



• SUMMARY OF THE ROUNDTABLE 

• Closing Comments on the Second Roundtable on Outcome Measures in Child 
Welfare 

Patricia Schene, Children's Division, American Humane Association 



• APPENDICES 

” Final Agenda 

- Speaker List 

- Facilitator/Moderator List 

- Planning Committee List for the Second Annual Roundtable on Outcome Measures 

- Participant List 



ACKNOWLEDGMENTS 



The support and participation of many talented individuals and organizations resulted 
in the success of the Second Roundtable on Outcome Measures in Child Welfare 
Services. American Humane Association (AHA) thanks the National Association of 
Public Child Welfare Administrators (NAPCWA), an affiliate of the American Public 
Welfare Association (APWA), for their leadership and ongoing commitment to co- 
convening the series of Roundtables. 

We thank members of the national Planning Committee for their guidance and 
direction in developing the agenda, as well as to those members who contributed to 
the development of the background document: A Framework for Outcome Measures 
in Child Welfare Services, prepared In advance of the Roundtable. AHA and NAPCWA 
are especially grateful for the presenters and facilitators who contributed their time 
and ideas at their own expense. Their work was a key foundation to the Roundtable's 
success. 

We also extend our gratitude to Lockheed IMS Child Welfare Services, Unisys 
Corporation and the Texas Department of Protective and Regulatory Services who 
generously supported the Second Roundtable. 

On behalf of all of our partners and colleagues, we are pleased to present this 
Summary of the Proceedings. 



Nancy C. McDaniel, MPA 
Senior Program Analyst 
Children's Division 
American Humane Association 




1 



INTRODUCTION 



The Second Annual Roundtable on Outcome Measures in Child Welfare Services was 
organized and convened by the American Humane Association (AHA) and the National 
Association of Public Child Welfare Administrators (NAPCWA), an affiliate of the 
American Public Welfare Association. This Roundtable, which represents the second 
in a series of five annual Roundtables planned by AHA and NAPCWA, was held in San 
Antonio, Texas on April 7-9, 1994 at the St. Anthony Hotel. 

The overall purpose of the Roundtable process is to provide an ongoing forum to 
collectively address the issues associated with the design, implementation, and 
utilization of meaningful client outcome-focused measures of service effectiveness. 
The goals of the second Roundtable were to: 

• review and select key outcomes appropriate for the children and families served 
by our child welfare systems; 

• build understanding regarding a conceptual framework for identification of key 
outcomes; and 

• explore processes for building agency and community consensus on key 
outcomes and indicators. 



Through panel discussions, presentations, and work group sessions, the Second 
Roundtable represents an ongoing effort to share knowledge and information and to 
develop a common agenda and measure the achievement of outcomes in child 
welfare. 
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Second Annual Roundtable on Outcome Measures in Child Welfare 

Welcome and Opening Remarks 
Kittie K. Arnold, M.S.W. 

I am pleased and honored to be here today and wish to welcome you to the timely 
Second Annual Roundtable on Outcome Measures in Child Welfare. I do so on behalf of 
Nancy Rawlings, current president, and the National Association of Child Welfare 
Administrators Executive Board. 

NAPCWA, just having celebrated its tenth anniversary, was founded as a mechanism 
whereby child welfare administrators could create an agenda for developing progressive child 
welfare policy and shaiing program information. In 1988, the Board approved Model 
Guidelines for Child Protective Services. 

In 1991, members of NAPCWA were involved with the development of American 
Public Welfare Association’s, Commitment to Change , which develops a framework for the 
delivery of services for childien and famili es. Also in 1991, the first forum on outcomes 
was held, at which tim e public child welfare adininistrators began to look at how to define 
success in child welfare programs. In 1992, this was followed by American Humane’s 
National Policy Institute, which supported the need to define the role of child welfare within 
the larger system of children and family services. 

NAPCWA is also involved in a collaborative effort with the State Mental Health 
Representatives for Children and Youth (SMERCY) to develop guidelines for working 
together with the community mental health system to provide services to children and 
families. 

As we enter this new direction in public child welfare, we can no longer do business 
as usual. We need to follow programs tlwt we know work; we need to define where we are 
going. There is a need for accountability, especially in this time of shrinking and competing 
resources. We must be able to define clearly what we are doing and find out if it is working. 

Historically, public child welfare has been reactive in our approach. In the 1950s and 
1960s, we were family-service oriented, providing services to needy families. Then, with 
the information defining clearly the results of parental abuse, the funding and programs were 
focused on child protection. We defined our programs by the number of investigations that 
were done. Few resources were available to provide remedial services, other than foster care 
and adoption. At the data began to show large numbers of children in care, the focus was 
shifted to a concern that children were being lost in the in the foster care system. This was 
followed by a large investment in resources to monitor processes and to review paper trails - 
the assumption being that if specific activities were done, the results would be good. 

I see this thinking as similar to the study just completed showing that bussing had 
little impact on the quality of education that children received, despite the premise that this 
activity in itself will result in a better quality of education for all children served. 

We now have guidelines for the direction we want to follow. We clearly see the 
results of our adaptive reactions to underfunding and the resultant fragmentation of services. 
It is our responsibility to develop goals that can be understood and can be measured, so that 
everyone will understand what we are doing, as well as how well we are doing. We need to 
be able to define our role in the community as a stakeholder with specific services we can 
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offer. Public Child Welfare agencies need to be able to articulate what we do; why we do it; 
and how many resources are needed to accomplish these ends. 

We can no longer continue to "do good"; to be evaluated by how the paperwork is 
done; to be assumed to be the protector of all children and show no progress for our "hard 
work." We currently are spread so thin and must often rely on removal as our only choice 
of intervention. This, in itself, has not been successful. We need to move beyond a 
description of processes to actual outcomes which will speak to what we expect to 
accomplish. 

This is the charge that has brought us together for this important Roundtable on 
Outcome Measures. We will hear from presenters the results of their work in this "cutting 
edge" arena of outcomes for child welfare services. We are honored to be a co-sponsor of 
this event, along with the American Humane Association and the American Public Child 
Welfare Association. 
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The purpose of this background paper is to introduce the participants of the Second Annual 
Roundtable on Outcome Measures in Child Welfare Services to key concepts and core 
outcomes ** to establish a common language and conceptual framework to facilitate our 
learning and participation at the Roundtable. 

The ideas and outcomes which are presented in this document grew out of our discussion at 
the first Roundtable and the involvement of a national planning committee made up of 
researchers and child welfare professionals. 

Last fall, in preparation for the Second Annual Roundtable, the American Humane Association 
convened a day long meeting of members of NAPCWA's Outcome Measures Committee along 
with chairpersons of the ad hoc committees formed at the first Roundtable. Committee 
members sought to: 

• define the goals for the Second Roundtable, 

• discuss hew to best plan the agenda to meet those expectations, 

• agree on a common conceptual framework to describe outcomes, and 

• define target outcome categories. 

The committee agreed that a written paper would be helpful in bringing everyone to a 
common starting point of discussion at the Roundtable. This working document is not 
intended to be a definitive or conclusive statement, but rather, a vehicle through which to 
build consensus by reflecting areas of general agreement. 

The background paper is adapted from topical papers prepared by members of the planning 
committee and reflects their exceptional talents and generous contribution of time. We wish 
to acknowledge the work of Dennis Orthner in preparing the narrative description of the 
conceptual framework. This provided a core document for extensive review, adaptation, and 
editing ~ reflecting the dynamic and evolving ideas of the planning committee. 

We are grateful for their ongoing support and commitment to building our knowledge and 
understanding of what it means to focus on outcomes in child welfare services. 
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I. TOWARD A FRAMEWORK FOR CHILD WELFARE OUTCOMES 

I 

I A. Introduction 

4 

The need for research on child welfare-related issues has never been greater. Incidents of 
child abuse and neglect continue to mount as well as questions regarding the effectiveness 
of current intervention strategies. All too often, research on child welfare has focused on the 
program or agency itself and the nature of the intervention rather than the consequences for 
children and families — process, not outcomes. 

Attempts to develop outcome-based research on child welfare interventions have tended to 
focus on the safety of children. Clearly, that is a primary goal of child welfare services and 
it should remain the focus of most research. In a classical research paradigm, the 
effectiveness of an intervention would be gauged by its ability to reduce subsequent child 
mortality, morbidity or incidents of confirmed maltreatment. However, this research model 
tends to focus on the reactive aspects of child welfare services and typically fails to account 
for, or encourage, the development of secondary or primary prevention efforts that are 
desperately needed in most communities in order to reduce the number and severity of long- 
term child welfare cases. 

This outcome evaluation strategy also tends to neglect many of the intervening variables that 
need to be incorporated in an intervention, especially those that involve the use of other 
support systems or that improve overall family functioning. Limiting or concentrating research 
on a narrow range of outcomes tends to focus interventions solrly on those outcomes. 
Broadening the scope of outcomes encourages the development of a wider range of support 
services in child welfare interventions that can better address a range of needs of children and 
families. 

The need to expand the scope of child welfare outcomes was particularly evident at the 
meeting of the First Annual Roundtable on Outcome Measures in Child Welfare Services, 
presented by the American Humane Association and the National Association of Public Child 
Welfare Administrators, an affiliate of the American Public Welfare Association. The 
Roundtable was convened March 11 -13,1 993, in San Antonio, Texas and co-sponsored by 
the Texas Department of Protective and Regulatory Services. Participants included child 
welfare policy makers, program administrators and researchers. When the participants were 
asked, in small groups, to define the potential range of child welfare outcomes, the list was 
extensive. It included consequences for children, their parents, foster parents, agencies, 
communities, and funders. Also included were a broad range of outcomes not specifically tied 
to child safety. Many of these outcomes incorporated the developmental needs of children 
as well as the potential benefits for family continuity and healthy family functioning. 

When the input from the groups was compared with the practice models in child welfare, 
however, it became clear that many of the outcomes proposed did not necessarily fit within 
a narrowly-defined set of interventions. Without a framework to incorporate the range of 
potential child welfare outcomes, there was a tendency to ignore the full potential for child 
welfare interventions. This lack of congruence between the range of potential outcomes and 
existing practice models required that we b>'oaden the scope of our conceptual framework to 
more accurately reflect the alternative str Regies for reform of the child welfare system. 

The purpose of this paper is to propose a framework for child welfare outcomes that 
incorporates a broader range of target categories of outcomes that can be used to guide 
research and intervention efforts. This framework is not meant to diminish attention -to the 
concerns for child safety, but to increase opportunities for child welfare interventions to focus 
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on other needs as well. Overall, it is proposed that broadening the scope of child welfare 
outcomes will promote a range of service interventions and greater interagency collaboration 
around child welfare issues and concerns. 



B. Dimensions of Child Welfare Outcomes 

There appear to be two major dimensions along which child welfare outcomes can be derived. 
The first dimension represents the target categories of outcomes -• conditions or behaviors 
that should occur as a result of child welfare interventions. The second dimension represents 
the focus of change. 

The focus of change for the child welfare agency is primarily that of children and families. 
Child welfare agencies, however, are one part of a larger system of response; and positive 
outcomes of an effective system of services to protect children and support families are 
reflected at the community level. Thus we can consider and define desired outcome 
indicators for each focus of change -- the child, the family and the community. 



Target Categories of Child Welfare Outcomes 

Child welfare services are typically directed to improve the conditions of children and families 
in order to e'^sure that children are safe and families are able to provide an environment for 
their continued nurturance. In addition, there are a host of developmental outcomes for both 
the child and family that also need to be considered Overall, there are four target categories 
of outcomes for child welfare. These include: 

• child safety, 

• child functioning, 

• family functioning, and 

• family continuity/family preservation. 

Child Safety 



The primary concern of the child welfare agency is child safety and the success of most child 
welfare intervention efforts is gauged by the ability of the family, child welfare agency, and 
community to protect the child from harm. Interventions and research have focused primarily 
on this factor, given society's interest in protecting its children and youth. This concern is 
long-standing and other efforts should not diminish the importance of outcomes related to 
protecting children from harm. 

Child Functioning 

Less attention has been given to the quality of life for the child, as well as, healthy and 
appropriate development. However, that concern is also one of significant interest. Healthy 
social, emotional, and cognitive development is essential to the welfare of children. Since 
research has demonstrated that children who have been abused and neglected are more likely 
to have difficulty with attachment later (including the increased risk of subsequent abuse and 
neglect of their own children), protecting children's safety without considering their other 
developmental needs is a somewhat narrow role for child welfare services. 
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Family Functioning 



While healthy family functioning is valued by child welfare services, limited attention has been 
given to promoting family functioning as a way of reducing reported abuse or neglect. Indeed, 
services to support family functioning have been very limited and coordination with other 
responsible agencies or departments has not always been prioritized. Indeed, healthy family 
behaviors and attitudes are a major factor in successful prevention efforts, and building or 
rebuilding these healthy patterns is necessary in order to protect children from further abuse 
and neglect. 



Family Continuitv/Familv Preservation 

Preservation and continuity of the family is another major target of child welfare services. 
Intensive efforts to provide family preservation services and support services to promote the 
reunification of families have been given significant attention over the last decade. However, 
there is also the need to respect and foster family and kinship ties whenever possible, even 
j when the child cannot remain safely in the home. The federal government, advocates, and 
child welfare professionals have increasingly recognized that the best place for children to 
grow up is in a family. They have also have advocated for significant resources to promote 
family preservation along with family continuity. 



Focus of intervention in Achieving Child Welfare Outcomes 

Traditionally, the greatest attention in child welfare has been on outcomes for children and 
families. However, it is essential to recognize the vital support that communities provide to 
children and families. In defining outcomes, there is increasing recognition that child welfare 
outcomes are needed at the community level as well as at the child and family level. 

Children 



A focus on positive outcomes for children has been the hallmark of child welfare. Services 
are targeted to protect children from physical or emotional harm and offer children 
opportunities for healthy development. Because of their dependency, children require services 
that can anticipate their needs and protect them from others who may take advantage of their 
vulnerability. Producing positive outcomes for children is not the sole responsibility of the 
child welfare agency. Indeed, coordination with other agencies is often essential to insure 
that children are receiving an array of appropriate services. 



Parents and Family 

Another focus of child welfare outcomes lies within the family itself. Families who have the 
psychological, relational, and economic resources they need are less likely to abuse their 
children. Therefore, child welfare research and intervention should incorporate outcomes that 
measure the ability of the family to promote healthy development and safety. It is not feasible 
to expect agencies to respond to all of the needs of children or to operate in loco parentis; 
parents must be empowered to understand their own needs as well as the needs of their 
children to ensure a healthy environment for growth and development. 
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Community 



The community-at'large also must be considered as a focus of potential child welfare 
outcomes. Clearly, the community is a major source of information regarding incidents of 
child abuse and neglect and compulsory reporting requirements affect community 
responsibility for children and families. Also, the community includes a variety of public and 
private services and resources that impact children and families. 

The community also shapes (as well as reflects) the environment within which children and 
families live. Parental values and expectations are often set or influenced by the community 
in which they live. Nationally, we have witnessed growing recognition that efforts to improve 
outcomes for children and families must incorporate positive community involvement in child 
and family development. 



C. A Framework for Child Welfare Outcomes 

Based upon the previous discussion, it would appear that a typology of child welfare 
outcomes must incorporate the two dimensions -- target categories of outcomes and focus 
of change. 

The following, Rgure 1 : A Framework For Child Welfare Outcomes, suggests that outcomes 
can be classified according to the four categories of conditions of the child or family and the 
focus of change, be it the child, family, or community. 

Based upon a review of the literature, it would appear that the majority of child welfare 
outcomes examined thus far focus on the child and target child safety conditions. Clearly, 
these outcomes are essential and are key sources of accountability for the child welfare 
delivery system. Still, as the framework in Figure 1. suggests, there are twelve potential 
outcome domains that the child welfare system could take into account in developing its 
services and evaluating its effectiveness. 

If the system of child welfare is to indeed incorporate within its scope the "welfare of 
children," it needs a broader examination of outcomes to which it can contribute. 
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FIGURE 1. A FRAMEWORK FOR CHILD WELFARE OUTCOMES 
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li. TARGET CATEGORIES OF OUTCOMES IN CHILD WELFARE 



A. Policy and Social Mandate to Achieve Outcomes 
Federal Social Legislation 

Several key pieces of federal legislation guide federal and state policy mandates in child 
welfare. Public Law 93-247, the Child Abuse Prevention and Treatment Act passed in 1974, 
provides funding to states to improve their child protective services systems and establishes 
greater uniformity and consistency between states regarding the reporting of suspected 
maltreatment. The Child Abuse Prevention and Treatment and Adoption Reform Act (P.L. 95- 
226) expanded the provisions of P.L. 93-247 and also addressed child pornography and 
adoption law reform. 

The Indian Child Welfare Act, passed in 1978, returned jurisdiction of Native American 
children to the tribes and addressed placement priorities for Native American children. The 
passage of P.L 96-272, the Adoption Assistance and Child Welfare Act of 1 980, required that 
states exert reasonable efforts "to prevent initial i.,acements and to provide efforts toward 
reunification of children with their families once the children are placed. The intent of this Act 
is to preserve families and to provide for permanericy and stability for children." (Guidelines 
for a Mode! System of Protective Services for Abused and Neglected Children and Their 
Families, 1988, Washington, D.C.: National Association of Public Child Welfare 
Administrators, pg. 17.) 

With the recent passage of the Family Preservation and Support Services; Omnibus Budget 
Reconciliation Act of 1993, (P.L. 103-66) the language and intent of federal policy is now 
accompanied by federal financial participation in support of an array of family support and 
family preservation services. 

Social Policy Mandate 

The social mandate for services to children and families is reflected in the notion that our 
society is based upon strong and healthy nuclear families. Virtually all of our federal social 
legislation either assumes or is intended to promote intact families. Although there is not 
agreement on a definition for "family," there is a high degree of consensus on how important 
families are to children. Children do best growing up in families that can safely nurture them. 
Even children who cannot safely reside with their parents benefit from continuity of family -- 
their models and lessons have lifelong impacts. The preservation of family ties, roots, and 
culture are all important aspects of family continuity. 

Although there is a fundamental social value supporting the rights of families and the 
responsibility of helping agencies to preserve and maintain families wherever possible, there 
is at the same time a growing, and sometimes conflicting, belief in the rights of children. 
There continues to be an ongoing struggle between these values and a lack of clarity on 
where the balance lies. Nevertheless, outcome measurements must primarily and consistently 
reflect the key social values of child safety and family preservation. 

Also, when we consider the four target outcome categories of child safety, child functioning, 
family functioning, and family preservation/family continuity, our minimal expectations are 
most clearly defined by policy and social mandate for child safety and family preservation. 
Minimal and acceptable community standards are not as clearly defined in policy for expected 
outcomes in terms of child and family functioning. 
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B. Linkages to Other Community Agencies 



Linkages between community agencies are an essential component in an effective system of 
response to abused and neglected children and their families. Child abuse and neglect 
problems, and the support and preservation of families, do not lend themselves to simple 
treatment approaches rendered unilaterally by a single actor. Informed by one discipline. The 
multiple and complex problems of children and families require intervention and treatment that 
is generally beyond the scope and expertise of any single discipline. Maintaining linkages 
between community entities enhances the quality of services provided at the community level . 

A significant component of child welfare casework services is directed to accessing basic ^nd 
supportive services such as medical, health care, educational, financial resources, housing and 
mental health. This requires essential agency linkages between the child welfare system, 
community agencies, law enforcement, and judicial courts. 

Indeed, when services from a child welfare agency end, the families are likely to have issues 
remaining that don't pose direct risk of harm to the child, but that need resolution through the 
provision of other services or resources in the community. A family's appropriate use of 
commonly available community services and resources also can be viewed as a positive 
outcome. 



C. Illustrative Outcome Indicators 

When defining goais and desired outcomes in child welfare, the safety and functioning of the 
child must be paramount and considered in conjunction with the continuity and functioning 
of the family. All outcomes must also consider the value of timely decisions regarding 
permanent homes for children. The outcome indicators within the target categories listed 
below are not intended to be exhaustive, but are meant to be illustrative and encourage 
discussion regarding additional outcome indicators. 

Target Outcome Category 1 . CNId Safety 

The goal for child safety is to protect children from harm, prevent further neglect or abuse, 
and reduce the risks to their safety or well-being. Examples of outcome indicators for the 
child, family, and community include: 

Child 



• Decrease in number of subsequent confirmed cases of abuse and neglect in cases open 
for child welfare services. 

• Decrease in number of subsequent confirmed cases of abuse or neglect within a 
specified period of time following the provision of services. 

• Decrease in number of confirmed cases of abuse or neglect of children in out-of-home 
placement. 



Family 



• Increase in use of appropriate discipline by parents vyith their children. 

• Improve level of knovyledge by parents regarding physical and developmental needs of 
children. 

• Decrease unnecessary disruption of families during the investigations of reports of 
abuse and neglect. 



Community 

• Increase in the number of reports of suspected child maltreatment in compliance with 
the law. 

• Increase in the availability and adequacy of an array of services for children and 
families who have been reported or confirmed for child maltreatment. 

• Decrease in the rate of adolescents adjudicated for delinquency. 

• Decrease in the rates of criminal arrests or criminal charges for child welfare cases. 

Currently, it is the responsibility of the public child welfare agency to respond to reports of 
child maltreatment and intervene when necessary to protect children from harm as a result 
of abuse or neglect. However, there is growing recognition that the responsibility for 
protection of children from harm is not solely that of the child welfare agency, but must be 
shared by other community agencies and actors. 



Target Outcome Category 2. Child Functioning 

The underlying goals for child functioning recognize that children have essential needs for care 
that go beyond child safety. Society is responsible to ensure that the normal development 
"d long-term emotional and physical health of children must be ensured by minimal standards 
uT care. These stanc 3rds relate to basic nutrition, clothing, shelter, health, education, and 
vocational needs along with standards addressing educational, emotional, and psychological 
development. 

Examples of potential child welfare outcome indicators in this area include: 

Child 



Improved school attendance and/or performance for children receiving child welfare 
services and within a specified period of time following the provision of services. 

Improved developmentally appropriate behavior and growth for children receiving child 
welfare services and within a specific period of time following the provision of 
services. 
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Family 



• Improved ability of families to ensure that the educational, developmental, and 
emotional needs of their children are addressed. 

• Improve the developmentally appropriate response by parents or families to their 
children. 



Community 

• increase in the availability and affordability of child care options for children who were 
the subject of a report of maltreatment. 

• Increase in the number of adolescents successfully involved in appropriate 
emancipation as demonstrated by involvement in educational or vocational activities 
in the community. 

• Decrease in the number of adolescents adjudicated for delinquency. 

Clearly, the developmental needs of children are influenced by abuse and neglect. These 
needs should be addressed immediately after child safety is addressed. Failure to consider 
the child's developmental needs can, and often does, result in significant problems later. 
Whether the developmental services are offered in the home, in foster care, or some other 
setting, adequate child functioning must be considered if child welfare services are to be 
successful. 



Target Outcome Category 3. Family Functioning 

The goals of improved family functioning are to provide services that will support and enhance 
parents' and families' capacity to safely care for and nurture their children. The implicit 
assumption is that improved functioning of the family will result in improved outcomes related 
to child safety and family preservation/family continuity and contribute to positive long-term 
outcomes. 

Sorhe of the potential child welfare outcome indicators in this area include: 

Child 



• Decrease in the level of child's fearfulness toward the parent. 

• Improvement in the quality of interactions between child and parent. 

Family 

• Improvement in families' use of effective communication and non-abusive techniques 
to resolve family conflict. 

• Improvement in families' sense of family cohesion. 

• Improvement in parents' capacity to provide adequate care, nutrition, hygiene, and 

supervision to their children. 
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• Improvement in parents' capacity to identify and access services in the community to 
meet their needs and contribute to their sense of well-being. 

Community 

• Increase in the availability and adequacy of recreational services and community 
supports to encourage healthy family functioning. 

It is not enough that child welfare agencies promote child safety; they must also support 
efforts to promote the healthy functioning of families. Keeping children in, or returning them 
to homes without better patterns of parental behavior increases the risk of subsequent abuse 
and neglect. Efforts need to be made to promote positive interactions between parents and 
children and between families and their social support systems. Again, these efforts often 
go beyond the scope of the child welfare agency, but careful collaboration across agencies 
can result in integrated programs and services that support healthy family functioning. To the 
extent possible, these services should also be the focus of primary and secondary prevention, 
targeting the range of families who may be at risk for problems in effective parenting. 



Target Outcome Category 4. Family Continuity/Family Preservation 

The value and importance of keeping children's ties with their families is reflected in this goal 
area. Family-centered services are intended to achieve safety for children by strengthening 
family and child functioning. When children must be removed from the home to ensure their 
safety, this goal area emphasizes the importance of maintaining family and kinship ties 
whenever possible. 

Some of the potential child welfare outcome indicators in this area include: 

Child 

• Increase in the number of children placed safely and appropriately with family or 
kinship relations. 

• Increase in the number of children placed in permanent homes within one year of initial 
out-of-home placement. 

Family 

• Increase in the number of families who maintain scheduled visits with children who are 
placed in out-of-home placements. 

• Improvement in the quality of parent/child interaction and satisfaction with the 
relationship for those children reunified with their families. 

Community 

• Increase in the number of adoptive and foster care homes that reflect the cultural and 
ethnic makeup of the community. 

The importance of promoting family continuity cannot be overstated. To the greatest extent 
possible, child welfare services should provide every opportunity for parents and children to 
remain united or to be reunited in as short a period as possible. In many cases, this also 
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means promoting the continuity of the parental relationship as well as other family 
relationships that are deemed appropriate for the rearing of children. Every effort must be 
made to promote continuity among families to provide a healthy environment for their 
children. 



D. Responsibilities for AcNeving Outcomes 

This Background Paper and the framework illustrated in Figure 1 . describes child welfare 
outcomes along two dimensions, target outcome categories and the focus of change. 
However, attaining these outcomes at the child, family and community level also implies 
responsibilities -- for the parent, child welfare agency, and community. These responsibilities 
can be viewed as the processes and activities directed towards achievement of outcomes — 
the means to the end rather than the end itself. 

Parents have a responsibility to learn appropriate discipline techniques and provide adequate 
care and supervision of their children. The child protection agency has a responsibility to 
investigate all legitimate reports of maltreatment made to the agency in a timely and thorough 
manner. The community also has a responsibility - to undertake the necessary planning to 
establish an adequate array of services to ensure the safety of children. 



While the Second Annual Roundtable will focus primarily on the definition of a conceptual 
framework to describe and define outcomes for children, families, and the community, we 
recognize that the roles and responsibilities of the family, child welfare agency, and the 
community must also be clearly defined and will be a focus of our agenda for future 
Roundtables. 



ill. CONCLUSION 

The dilemma faced by participants at the end of the first Roundtable can be characterized by 
the assertion that the public child welfare system cannot be held responsible for achieving 
positive outcomes for children and families without the involvement of the larger community. 
"The child welfare system," it was stated, "inherits the failures of the larger community to 
commit to positive outcomes for children and families." 

It became clear that while there is no doubt that achieving positive outcomes for children and 
families is a responsibility that extends beyond the child welfare system, there is value in 
defining the child and family outcomes relevant to child welfare, along with outcomes at the 
community level. The conclusion then, was for the Second Annual Roundtable to identify 
target categories of outcomes relevant to child welfare and develop a framework for these 
outcomes that addressed the interconnecting focus on children, families, and the community. 



The framework that is outlined in Figure 1 and the discussion in this paper illustrates the four 
target categories of outcomes (child safety, child functioning, family functioning and family 
continuity/family preservation) for each focus area of change (child, family, and community). 
It will be our task during the Roundtable to generate a list and prioritize outcomes within each 
of the target categories. It is hoped that this process will further our common understanding 
and clarify the role of the public child welfare agency in achieving positive outcomes for the 
children and families it serves. 

We look forward to your thinking and contributions to this process at the Second Annual 
Roundtable. 
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I. Presentation hv Ann Kelly Anderson. Child Welfare A nalyst. State of Colorado 



I’m Ann Kelly Anderson and I am going to give you a quick overview of our 
workgroup, the work we have done so far, and what we plan to accomplish in the future. 

First, let me give you a quick overview of the agenda. David Bernstein will be talking about 
o^r Outcomes. We have Three Desired Outcomes and he wiU present some examples of the 
minimum standards for each of the Three Desired Outcomes. Terri Bailey from the Piton 
Foundation will present the Colorado Conceptual Model. Kittle Arnold will discuss key 
community indicators and community collaboration. Lloyd Malone is going to discuss some 
of our implementation issues. 

Background 

First, I want to talk about how the workgroup was formed and how we got involved 
in Outcome Measures in Colorado. A couple of years ago the Colorado State Legislature 
actually required us to provide to them a report on outcomes for children and families. They 
specifically asked for formation about length of stay, number of moves, and recidivism for 
children in Foster Care. These questions were quite sophisticated and specific for the 
Legislature. We provided some information from state level that was very important. It 
dispelled some of the myths and misinformation-information about the length of stay of 
children in Foster Care. We found out that many children, ^>proximately 60%, were in 
short term placements, i.e. placements lasting less than 90 days and that the majority, 82%, 
of children in all foster care returned home after placement. It was very powerfol 
information from the stand-point of providing actual data from our automated systems to 
establish fact over antidotal information that is often provided to the Legislature. 

The second event that has happened in Colorado is that we are in threat of a lawsuit 
and we are right now in the negotiation process with the ACLU and the Colorado Lawyer’s 
Committee. When we started the process with the ACLU, we believed that they would place 
great emphasis on outcomes measures and child-based outcomes. It turns out the majority of 
items in the lawsuit have more to do with process and coimting activities than outcomes for 
children and families. They’re not necessarily what we think of as outcome measures. But 
one of the things that fallen out from this process is now we are getting a lot more resources. 
I think we are adding somewhere around 300 new caseworkers in the counties. Now it is the 
State Legislature that is saying "Okay, we’ll give you additional resources to comply with the 
negotiated settlement but we want to know what are we going to get for that. What are the 
outcomes for children? Do adding aU those resources really make a difference?" So we got 
to outcome measures kind of through the back door in that process. 

Another important piece is that we are going through is a restructuring of state 
agencies. The Department of Social Services is combining with the Department of 
Institutions which includes the Division of Youth Services and the Division of Mental Health. 
T his involves significant changes in both state and local level organizations. New legislation 
requires local governments, local agencies, communities, and clients to get together and 
conduct local needs assessments, coordinate local planning efforts and pull resources to 
make it easier for families to access services and more efPrxtive. We are starting to talk 
about what the desii'ed outcomes will be in bringing these services together, and what the 
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result of this coordinated approach is having on families and children. Are they better off 
than before? So that is a r^y important piece. 

The last part is "PAC Ret^esign" as we call it in Colorado. This is referring to the 
Family Preservation Act and the new legislation. This legislation calls for outcomes for 
children and families. 

Also I’d like to teU you who got involved in this workgroup mitially and how we 
pulled together our group. The County Departments of Social Services really promoted 
getting this workgroup together. We provided state level information to the Legislature 
which was important, but counties ne^ed information that was broken down to the local 
level and more meaningful to counties. The counties felt it was important to start using 
outcomes to manage and evaluate services. So the counties really drove the workgroup 
formation. 

We also got the foundations involved. Nancy McDaniel from the American Humane 
Association is a regular working member of our committee. It has been extremely beneficial 
to have Nancy, especially with the tie-in to the national outcomes movement. She keeps us 
abreast of what is going on nationally. It is really helpful; she brings a broader perspective to 
our efforts. Terri Bailey from the Piton Foundation has also brought her research 
background regarding children and families. She is a wonderful thinker and communicator. 

One reason it is so critical to have foundations involved in this process, is that it neutralizes 
the boundary and political issues between the state and counties. It really has helped us set 
aside traditional state and county issues and move forward. 

Accomplishments to Date 

I am going to highlight some of our accomplishments so far. This includes products 
we have developed and tasks that we have completed so far. Many of these products we will 
be showing you later in our presentation such as the Model and the Three Desired Outcomes. 
We also developed a Glossary of Terms. That was a really critical piece for us because we 
had social workers and non-social workers, we had people from within the system and people 
outside of the system who didn’t know the system jargon, we had researcher types and non- 
researchers, and we found that we were aU using different terms. The glossary was really 
critical for us to find and use some common language. When we talked about inputs and 
what that meant, the glossary provided a critical break through in our early discussions, 
which were very conceptual. So that was really helpful. 

Another activity that has been helpful is the presentations we have made to different 
audiences. I think that has really helped us refine our thinking. Each time we make another 
presentation, we learn something and improve our plans and ideas. 

Quickly, I will highlight some of our future plans. We plan to pilot the Outcomes 
Model and outcome measures in several pilot sites. This will not include entire county 
operations, especially in the larger counties, but perhaps program or service areas within the 
counties or a service unit. We are developing pilot criteria and incentives for counties. 

We are also developing a research and evaluation plan. We are working with the 
University of Colorado, the School of Economics to conduct a comprehensive data 
evaluation. We are going to place a strong emphasis on technical assistance for the pilot 
sites and include community building training. We have developed a community data base of 
around 150 variables. We’re going to provide this information to communities and help 
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communities leam how to use that data to plan and find out what the gaps are on the 
surfaces. The other task that is really important is marketing. Yesterday, we were 
talking about the negative newspaper article regarding Child Welfare and we plan to work on 
a marketing plan with a public relations specialist from the Piton Foundation to better inform 
the pubUc and highlight some of our efforts and successes. We also mentioned yesterday 
that outcomes-based education is underway in Colorado. The term "outcomes" raises 
questions and issues by the public. We are trying to find words and ways to present this 
information that doesn’t get people on the wrong side of this issue. We’re going to be 
working on the public relations and marketing piece and make a real conscience effort to 
work through some of those hot topics. 

I am going to turn it over to David Bernstein who will talk about the three desired 
outcomes. 
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n. Presentation bv David Bernstein. Denver Countv Department of Social Services 
Three Desired Outcomes 

My part of the presentation is going to be to talk about three desired outcomes which 
are up here and to talk a little about some issues connected with that. We developed these 
three desired outcomes which are in many ways similar, part of this is because of the linkage 
we had to the National group to help plan this workshop through Nancy McDaniel. 

We developed three outcomes that are very similar to what was presented late 
yesterday by Nancy, but our model really doesn’t separate child, family and community. 

We are loo king at it more as an integrated approach throughout the whole process and 
f rankl y^ I also think that is going to foster more community ownership of the problem than 
single agency ownership. We also in our model, detail what we are looking at for minimum 
outcomes. We really have taken a look at trying to concentrate on behavioral measurements. 

If you look on the more complete schedule you will see things like family functioning 
scales. We believe that everything has got to be documentable and behavioral as much as 
possible. Referrals don’t fit as much as successful connections which result in service 
delivery. I will talk about that in a couple of minutes. 

There will need to be some synthesis of process and outcome measurement. That was 
clear yesterday from a couple of vantage points. But one of the things that we are going to 
need to do is very carefully address this issue. As a potential pilot county, I can tell you that 
Denver won’t pilot this unless we can be exempted from some of the incredible number of 
process measurements we are involved with now. That is not to say that the process 
measurements weren’t developed for good reasons as some of the discussion yesterday 
indicated, but we have got to understand that we are in a different era now and we are going 
to have to do things differently. I have said that three tunes. So we are going to need to 
deal with the fact that workers can’t take more accountability without some things being 
taken off of them and that’s going to have to be addressed. 

We all know that staff are overwhelmed. Let me just go through these three 
outcomes: 

1. The child is safe. 

2. The child will be safe from maltreatment. 

3. The safety will improve for children and the communities 

in which they serve. 

Understand that this is the crystallization of the process that took many, many months 
to develop and this is really the focus of all of our outcomes at this point. The family being 
preserved was another, that family continuity is maintained while minimi zing non-permanent 
placement and maintaining child safety. And finally that permanency will be achieved. That 
children will live in safe permanent homes respectful of the child’s cultural and kinship 
identity. Now those are very nice terms. Doing it is going to really require a tremendous 
amount of activity and collateral cooperation. You have the handout that goes through the 
details of the three outcomes that Arm was talking about. I wanted to go through and 
highlight a couple of these. Within the child is safe I have picked a couple. 
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Decreasing the percentages of cases with substantiated abuse and neglect within 12 
months after case closure. That means developing a tracking system for families that 
doesn’t stop the minute they leave our doors. Increase the percentage of cases referred to 
community agencies. Increase the percentage of cases that actually receive services from 
other agencies and reduce the percentage of the cases re-reported to social services for 
substantiated abuse or neglect. 

I’d like to talk about this one for just a second. The responsibility for succeeding 
with the outcome has to a lot with the reliance on community involvement. You can just 
make a referral and forget about it. What you are going to need to do is to address the 
barriers that stop when someone is referred, maybe by the child welfare worker to the mental 
health agency. What are the barriers to that person receiving those services? Is it distance? 
Is it the fact that the staff in a given agency are culturally incompetent? All of those issues 
are going to have to be addressed. 

A good example of positive change is to put the services in the schools or to put the 
services in a public health station where, at least, we have found in Denver, people are less 
inhibited; they are less frightened of those agencies, particularly if you have pt a good 
family resource school that has put a lot of energy into devel(^ing relationshq)s with their 
constituencies. Secondly, we are looking at families being preserved meaning increasing the 
percentage of children who remain safely in their own homes at six months or twelve months 
after closure. This involves data we in Colorado do not have because ours ends when the 
case is closed. 

Increase the percentage of cases in which the child is returned home within 30 days 
after initial removal, and increase the percentage of placements made in the home community 
of the child. More again, this caimot be done in isolation. This is going to require a 
tremendous amount of work to get agencies to be more compatible with whom we serve. In 
Denver, we sit miles away from most of our communities in a centralized building and have 
very little connection with the neighborhoods we serve. We are trying to address that by 
putting more initiatives and staff out in the communities. Finally, Family Permanency being 
achieved. What that means is basically that we are going to increase the percentage of 
children we place with families or friends. We have more relative evaluators in my agency 
than we have ever had before. Our goal is to decrease the amount of time between the 
beginning of the first placement and placement in a permanent home and to increase the 
percentage of permanent placements that are maintained twelve months after initial 
placement. Here again, community support will be crucial. Churches are an excellent 
respite source for a lot of families, particularly families in minority communities. We are 
going to do a lot of things with other agencies in order to achieve these outcomes. 



in. Presentation bv Terri Bailey. Piton Foundation 



Introduction 

Before I present Colorado’s model to you, I would like to make a few introductory 
comments. I work for The Piton Foundation, a private, non-profit operating foundation 
located in Denver which focuses on issues of poverty. We work at a number of levels 
including state and federal policy reform and as such, have had the opportunity to work on a 
number of reform initiatives ranging from education reform and health care reform to 
welfare reform. I must say that when I was asked to participate in the Colorado Child 
Welfare Outcomes project, I {^reached the project with a degree of cynicism. What had 
distinguished the reform efforts that I had previously participated in was that key players did 
not necessarily rerogime the need for change and many, in fact, were dragged Mcking and 
screaming to the table. Reform initiatives then became lengthy arduous negotiations in which 
everyone worked hardest to protect their turf and to maintain as much of the status quo as 
possible. 

My experience with the Child Welfare Outcomes Project in Colorado has been quite 
the opposite. Everyone at the table comes willingly. All are deeply committed to change and 
improvements m the child welfare system. And all view the child and family as their first 
priority, not protecting turf. It has meant that as risks were encountered by participants that 
would normally have stalled the process, we were always able to move on by asking what 
would be best for the children and families child welfare systems are intend^ to serve. 

There has been no issue this group has not been willing to face squarely. It has been an 
honor to work with this group and to experience ’reform’ as it perhaps was always intended 
to be — creative, energetic and focused on the consumers of services, not the providers. 

Developing our model in Colorado was a difficult task that stretched all of our 
learning and loiowledge. One of the first things we had to do as a group was to identify all 
of the things we did not know which was ALOT. To help us move forward, we brought in 
Madeleine Kimmich of the Human Services Research Institute of Maryland. Madeleine had 
spent a great deal of time thinkin g through the same questions we were now struggling with 
and was able to help us identify and label pieces of the puzzle and to provide a sense of 
structure that allowed us to move forward in develt^ing the Colorado model. 

The Colorado Model 

In this part of the presentation, we will be walking through the outcomes model 
developed for Colorado (exhibit 1). As Arm discussed, the outcome measures we developed 
all fall within three priority areas. We selected one of those — the child will be safe -- for 
today’s example. In designing our model, we specifically intended to arrive at a model that 
could be used to develop an individual case plan for a specific child or family. We also 
intended that the same model could be used to guide agency planning and program 
development on a larger scale. And we specifically intended that this same model be used for 
whole communities to come together around the issue of protecting children from 
maltreatment. As John Mattingly from the Annie Casey Foundation said yesterday, outcome 
measures need to operate within four simultaneous contexts: public policy, program 
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management, operational, and program evaluation. While we hadn’t been able to describe it 
quite that clearly, those are exactly the contexts in which Colorado’s model has been 
designed to reside. 

As we began our deliberations, not surprisingly, we started with outcomes. But 
progress was slow because there were so many exceptions. How could different communities 
be held accountable to the same outcome expectation if the services they provide or the 
manner in which they provide them differ drastically? How can we account for the 
differences between communities in terms of resources and need? How do we protect against 
the risk of having our measured outcomes show up as front page bad news in the local 
newspaper? To provide opportunities to answer these and other questions, our model is 
broken into four distinct areas. 

A. Input Indicators 

The first area in the Colorado model is ’inputs’. In our glossary of terms, inputs are 
defined as, ’the capacity to provide effective services’. Input indicators are critical for 
understanding what are achievable outcomes for whom and under what circumstances. Inputs 
include internal factors such as descriptive information about the child and the agency but 
also includes external factors such as the availability of community resources. What you’ll 
see here is an attempt to strike a balance between those internal and external factors all of 
which impact the capacity of an agency to achieve desired outcomes. 

The first area of Liputs is child and fanuly. Examples of inputs might include a 
description of the level of abuse or the quality of the parent/child relationship. Is family 
liv ing in severe poverty which might limit their ability to engage in certain activities? What 
are the family’s demognq)hic characteristics? What is a family’s distance from other service 
providers? All of these can be gathered at an individual level to help guide the development 
of an individual case plan but can also be gathered at an aggregate level to help guide agency 
program development and management by understanding the characteristics of persons 
needing services. Frankly, these are questions caseworkers routinely ask as they work with 
individual families but agencies rarely ask these questions as they engage in program 
development and evaluation. 

The second area of inputs is agency characteristics. What resources does the agency 
have available for child welfare services? What is the caseworker/ supervisor ratio? But, in 
addition, does the agency’s general philosophy support an outcomes approach to child 
welfare services? Will the county commissioners be supportive? 

The third area of inputs is community characteristics. If you are attemplmg to provide 
services in co mmuniti es with high unemployment, high poverty and not surprisingly, high 
abuse and neglect rates, you might find your capacity to provide effective services very 
different from the capacity evident in communities with few of these problems. The services 
you provide and the outcomes you anticipate as a consequence would also be vei^ different. 
Examples of community inputs include the socio-economic characteristics of a given 
community but also describe a community’s resources. Does the community have a history of 
collaboration? Someone mentioned yesterday that the task we are erigaged in is not one we 
can do alone. Understanding this and recognizing that protecting children is not the sole job 
of the child welfare agency but rather one that belongs to the •entire community is a hallmark 
of the Colorado model. 



Process Indicators 



Process measures are the ’description and measurement of how the service or 
intervention is implemented and delivered rather than what changes following the 
intervention’. In Colorado as in many other states, process measures are what the child 
welfare system is currently held accountable to. This emphasis is reinforced by events such 
as the lawsuit which imposes additional process; more paperworic, filling in more boxes, 
adding additional checkpoints. Many hope that a reliance on outcomes will replace the 
current process focus. However, in Colorado, we view outcome measures not as replacing 
process but rather as moving beyond process. Process measures are still needed but what is 
needed as well is the knowledge of the results of process; what happens as a consequence of 
the services and interventions delivered. In this regard, process measures are critical to the 
model because they allow us to test, validate and change our services and interventions to 
make the most of our inputs. If we do not know how process relates to achieving good 
outcomes, we ultimately will not do a good job of selecting the process measures that tell us 
the most. We are doomed to settle for greater and greater numbers of outcome measures if 
we never identify the relationship between process and change. 

As evident in the discussion about inputs, there are two levels of process relied upon 
in this model: agency services and community services. Examples of agency process 
measures will probably include the completion of an assessment and what that assessment 
should include, ensuring that rqxsrts are responded to in a timely fashion, the development of 
a treatment plan. Examples of community process measures might include the development 
of collaborative agreements between public and private agencies for the provision of key 
services such as substance abuse treatment or mental health, or using a team interagency 
approach to staffing new cases. All of these would tell us how an agency or community plans 
to go about delivering services and appropriate interventions. 

C. Output Indicators 

The Colorado glossary defines outputs as, ’the products or services delivered by an 
agency or its stafT. This is frequently defined in terms of units of service. These differ from 
process m that process tells us how and outputs tell us what. These are typically the things 
you expect to see once services are begun: has the treatment plan been initiated, is the family 
in compliance with the plan, has the court accepted the plan? This is where Colorado’s child 
welfare system currently stops. Have we done what we set out to do? However, in an 
outcomes model, it is not enough to know whether we did what we said we would do. We 
need to also know whether in so doing, we made a difference in the lives of the children and 
families under our care. 

D. Outcome Indicators 

Outcomes are the results or desired effects towards which programs, services or 
interventions are directed. Outcome indicators are the events or benchmarks that suggest 
progress towards attainment of a desired outcome. It is not what we do or even how we do it 
but rather why we do it that should guide all work: the belief that children should be kept 
free of abuse and neglect. In this model, examples of outcome measures might include 
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improvement in family functioning, decreases in the frequency or severity of abuse or 
neglect, or preferably the elimination of further incidents of abuse or neglect. These are 
measures that teU us whether we have lived up to our mission but also whether given the 
inputs we have to work with and the services we provide, were we effective in improving 
cMld and family outcomes. 

One final note about the Colorado model. You’ll note that the model is not as linear 
as it tq>pears at first glimpse. There are not only a series of arrows connecting inputs to 
process to outputs to outcomes but arrows linking outcomes back to inputs as well. We 
believe strongly that having achieved or not achieved the anticipated outcomes is not the end 
of the child welfare outcomes model, but rather the beginning. We do not yet know given 
our agency, child and community characteristics, what process or combinations of processes 
work best to achieve the desired outcomes. It is only after we’ve done our best and are 
willmg to take an honest look at what we have achieved that we will be able to better 
describe the combination of characteristics and events needed to achieve certain results. Even 
then, however, the world in which child welfare services operates is not static but rather 
constantly changing. Just when we think we understand the needs of children and families 
and the capacity of our respective communities to meet those needs, something happens and 
we need to go back to the drawing board. We have a change in elected officials in our 
community and all of a sudden we have more or less resources or a different political reality 
to deal with. A major employer closes down and the resultant unemployment and family 
stress accounts for an unanticipated increase in abus^ or neglected children. 

The model we developed will be tested to detemdne its’ ability to respond to just such 
changes. We developed the Colorado Child Welfare Outcomes model to respond to a very 
specific need -- the need for a planning and evaluation tool to routinely and systematically 
inform and influence the process of delivery of child welfare services. Just as the 
achievement of outcomes is only a beginning, this model represents a starting place for 
Colorado’s efforts to better respond to the needs of abused and neglected children. 
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County/State Outcomes-Based Child Welfare Management Project 

Major Milestones 

Phase L Conceptual Model Design 

* Formed work group comprised of state/counties/foundations. 

* Received training from national expert (September 1993) 

* Developed Vision Documents 

* Statement of Purpose 

* Mission 

* Guiding Principles 

* Glossary of Terms-common language 

* Developed conceptual model- inputs, process, outputs, outcomes 

* Three Desired Outcomes 

* Completed minimum standards for each desired outcome using our model 

* Community Level Indicators 

* Data Evaluation Plan with University of Colorado 

* Plan to Develop Child and Family Assessment Instrument with Denver University 

* Client Pathways 

* Pilot Criteria, Selection and Design Developed 

* Presented model to different audiences 





Three Desired Outcomes: 
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maintained whiie minimizing non-permanent 
placement and maintaining child safety. 




Desired Outcome #1: CHILD iS SAFE 



Child will be safe from maltreatment./ Safety will Improve for children and the 
communities in which they live. 

1.1 Child not re>abused / Other Children In The Home Are Not Abused 

♦ Decrease substantiated abuse and neglect in families open to Child 
Welfare Services. 

♦ Decrease the percentage of cases with substantiated abuse and neglect 
within 12 months after case closure. 

♦ Decrease substantiated abuse and neglect in out-of-home placement. 

♦ Decrease in severity of any incidents of subsequent abuse and neglect 
within 12 months of initial substantiated abuse. 

1.2. Safety of children and communities will improve. 

♦ Increase the percentage of unsubstantiated/unfounded cases that are not 
subsequently re-reported within 12 months. 

♦ Improve child and family functioning. {Instrument being developed}. 

♦ Increase percentage of unsubstantiated/unfounded reports referred for 
services to community agencies. Of these referrals, increase the 
percentage of cases who actually receive services from other agencies. 
Of those that receive services from other agencies, reduce the 
percentage of cases re-reported to social services and subsequently 
substantiated. 

♦ Decrease rates for criminal arrests or criminal charges for child welfare 
cases. Decrease in criminal activity of youth services cases. 

♦ Reduce adjudicated delinquency rates for children receiving child welfare 
services. 

♦ (For children in child welfare services) Increase the number of 
adjudicated delinquents that complete probation successfully. / Reduce 
rate of revocation of parole or probation. 

♦ Reduce risks/community stressors for abuse in areas of economic stress, 
family formation and structure, violence, and the absence of coping skills 
and resiliency factors. 
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Desired Outcome #2: FAMILY IS PRESERVED. 



Family continuity is maintained while minimizing non-permanent piacement and 

maintaining child safety. 

2.1 Child remains in own home. 

♦ Increase the percentage of chfldren who remain safely in their home 
within six months and twelve months of case closure. 

♦ Increase durability of intervention so issues of safety do not return to out- 
of-home placement within six months and twelve months. 

2.2 Quickly and safely return children to their families. 

♦ Increase the percentage of open cases where the child is returned home 
within 30 days after initial removal . 

♦ Improve child and family functioning. {Instrument being developed}. 



♦ Decrease the average number of days a child spends in non-permanent, 
out-of-home placement. 

2.3 Strengthen a child's connection to family and community. 

♦ Increase the frequency of contact between children in out-of-home 
placement and their families. 

♦ Increase the percentage of placements made i:i the home community 
when placement is appropriate. 

♦ Increase school attendance while child is in placement. (Decrease 
truancy of adolescents in out-of-home placement). 



O 
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Desired Outcome #3: PERMANENCY WILL BE ACHIEVED. 



Children will live in safe, permanent homes respectful of the child's cultural and 
kinship identity. 



3.1 The number of piacements/moves are minimized. 

♦ Decrease the number of times a child is removed from their home. 

♦ Decrease the number of unplanned moves while the child is in an out of 
the home setting. 

♦ Increase the percentage of adoptive placements without disruption prior 
to finalization. 

♦ Increase the percentage of permanent placements that are maintained 12 
months after the placement. 

3.2. Placements are in safe and least restrictive setting. 

♦ Increase the percentage of out-of-home placements that are of same race 
and ethnicity. 

♦ Improve family functioning /improve child well-being/reduce risks. 
{Instrument being developed}. 

♦ Increase the percentage of children appropriately placed with family or 
kinship ties. 

3.3 Quickly achieve permanent homes for children. 

♦ Decrease the time between the beginning of the first non-permanent 
placement and placement in a permanent home. 

♦ Increase the percentage of adoptive placements that are finalized within 
twelve months of adoptive placement. 

♦ Increase the percentage of children who have permanency achieved 
within 18 months. 

3.4 Equip children with "independent living" status with the necessary skills 

to live independently. 



Juation of Performance 








• Resources 
Colloborotive Efforts 



IV. Kittle Arnold - Arapahoe Countv Department of Social Services 



In going through the indicators, Terri referred to John’s speech around the public policy 
piece and I think that we have been willing, as a system, to own some things that we really don’t 
own as far as responsibility for things we have no control over. The quote by James Garbarino 
about violence in the community and about our community and about child abuse we thought 
was very appropriate. 

Terri said to look at what kinds of things we have learned from the research we’ve done 
in our area that directly impacts child abuse and neglect. Indicators of economic stress, 
indicators of family structure stress, indicators of coping factors and indicators of violent stress - 
- we’ve found that these were very much interwoven within our system and with the families we 
work with. The other thing that we discovered was how this played out in different communities 
and how you target resources or intervention in communities. It became very significant. Terri 
developed some maps of the state. These are our counties and this is the child abuse and neglect 
rates for 1992. And as you can see, there are the areas with the green which are between 150% 
of the Colorado average. The red areas are more than 150% of the Colorado average. And you 
can see those stand out fairly clearly in this map. In order to look at how those sort of coincide 
with each other, we also developed a map on poverty rates. As you can see some of the overlap 
has now occurred. This is free school lunch participation rates in 1992 which are indicators of 
poverty. The areas that are cross hatched are more Aen 150% of the Colorado average and then 
the ones this area is between 150% of the Colorado average. Again, this picks up some of the 
counties that were high in abuse as well as high in poverty. It also plays out that you will see 
those with increases in poverty or high poverty levels but don’t have high incidents of abuse. 

I think that that was a very interesting statistic or thing to note in that the counties that 
were fairly poor and did not have high rates of abuse and neglect were, and you can see some 
of those, the indicator is really an increase in poverty and the counties that have experienced 
strong increase in poverty and unemployment were counties tijdt tend to be higher in abuse and 
neglect than some of the very very poor counties in our state. 

One of the other community indicators we looked at was the teen birthrate. Now again 
you have this in and what you can see up here is Denver’s area, a very small area but it really 
stands out as far as alot of community indicators. This is teen birth 1992, again the cross match 
is more than 150% of the Colorado average and the other is more between 150% of the 
Colorado average. Again, we see some very specific counties that show up with the teen 
pregnancies. 

These are things that have not been explored as far as I am aware of within the state as 
far as any statewide planning on how you might devote resources or what kind of program might 
be specifically targeted for these areas, but we do know there is a correlation b^een that and 
the chid abuse rate. Some of the as I said don’t show up as strongly but I think it is a very 
interesting betrayal that really speak to the public policy pieces as well as program development. 
All four of those areas in lool^g at some kind of policy decision that would really effect the 
services that are delivered. 
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V. Presentation bv Llovd Malone. El Paso County Social Services 



I am going to switch gears here a little bit and talk about some implementation issues. 
The purpose of our conference is to present and discuss conceptual models and frameworks. 
However, I want to use a little of our tune today to talk about some of the effects of 
implementing this model in a corrununity. 

We are a part of an inter-agency collaborative effort in Colorado Springs to put a 
multi-agency program together that’s based partially on demonstrating what an outcome 
model can do. What happens when you work with a group of agencies and introduce a 
model like the one you heard about today, and what happens to the mission of child welfare 
when you do that? Our experiences are instructive. 

First of all let’s mention the agencies we are talking about: social services, mental 
health, public health, employment and tr aining , and a local school district. The outcomes 
we started out with were safety, permanency, reunification, and child functioning, all a 
reflection of what we do basically in child welfare. When we got done with the plarming 
process several months later, we agreed on these and several others as the "goal areas" or 
the outcomes that we ate going to work toward as a corrununity. So as a result of 
collaboration, we had an expanded mission and a more exhaustive number of outcomes to 
pursue. Certainly it has affected how we are going to deliver services in our public social 
services agency. So now we also have physical health, emotional health, a child who is able 
to select appropriate behaviors, family fonctioning and achievement around school 
performance vocation in addition to the original outcomes. We have indicators for each 
of these as weU as specific standards for our program. So, the model is affected by who is 
around the table and by the negotiation of the kinds of desired outcomes the corrununity as a 
whole will support. 

I want to give you another example of one of the opportunity costs of doing 
outcomes. In the last year, since May of 1993 up until February, I have compiled the 
amount of time it has taken to develop this project including the development of our 
framework. We took this Colorado model that you heard about today and adapted it to what 
was workable in Colorado Springs. So it was corrununity brainstorming, strategic planrring, 
etc. Finally I think in the end of October of this year, we firrally had our framework, a 
model that we could implement. But that had taken 800 hours of community time, seven 
other colleagues of mine and myself meeting about once every other week, anywhere from 4- 
6 hours a week to develop the model. That was one of the costs of outcomes 
implementation, and now we are talking about developing the children’s plans which is a 
very difficult process in itself. We have been testing the children, establishing baselines for 
them, compiling and entering new data, doing community development in the agencies we 
will work with, and developing traiiring packets for the people who are actually going to 
work the model. Developing a local model has cost 1,030 hours for all the people involved. 
After four months of serving clients at this point, we’ve put in 1,830 hours developing the 
model and changing the Colorado model to fit our corrrmurrity. 

Moving to implications around these issues, we thought it was important to pose some 
questions to you. They remain questions because I don’t think there are any sta n da r d i zed 
answers. We don’t have answers that will fit very neatly for every community or every state 
or even possibly in every neighborhood. These questions, I believe, are important for 
anyone looking at outcomes. The first one is around the control of input variables and the 



ownership of outcomes. 



• What does this mean for child welfare services to be subject to roles we don’t have 
now and to outcomes we don’t control? Obviously this question can be answered 
differently in every community. 

• How does the mission of child welfare change when we engage in community 
collaboratives? 

• How does the deflnition of outcomes change as a function of the population served or 
visa versa? This was a significant issue in our community: depending on who was 
around the table, the question became, "Well, who are we going to serve?" Then 
following that, people would ask "What are you trying to accomplish with your 
program? What are the outcomes you’re looking for?" People would then say, "Well, 
depending on what kinds of outcomes you choose, or I’ll need to choose a different 
population to bring to the program." 

So there was a real developmental aspect to the process for our community, but finally 
we decided that choosing the population, the actual clients to be served, had to come first . 
Very clear criteria for client selection drove everything else more clearly in our community 
than any other variable. 

Another thing is the opportunity costs issue. That is, when you implement this model, 
what do you pay out in terms of time, resources, and of course the avoidance other possible 
options you could have chosen if you did something differently? For example, now that 
we’ve got everything figured out in our program, the case managers who are working in this 
program continue to spend 4-6 hours a week just working on outcomes-related activities. 

I am going to move to implementation issues in context of standardization versus 
flexibility. There are variable effects on the development of programs and struggling with 
how you evaluate them as well as implement them. But I think it is very important for you 
to struggle with that issue. In our state we are wrestling with it not only from what is going 
to be done differently in one community versus the next, but also if we are going to do it 
differently what is the kind of research design we need in place to evaluate the different 
models? Not only did they work, but also what are the implications for replication? 

I think we have a few minutes for a few questions. 





Presentation Outline 

Lloyd Malone, El Paso County Department of Social Services 
Colorado State/County Outcome Workgroup 
Second Annual Roundtable on Outcome Measures 

I. Discussion of implementation in the context of the Child Welfare mission. 

A. Control of inputs/ownership of outcomes. 

• What does it mean to child welfare services delivery and program evaluation 
when we select outcomes which we don’t control? 

• How does the mission of child welfare change when we engage the community 
in developing community outcomes? 

B. Definition of outcomes as a function of the population served. 

• What effect does changing the "population to be served" have on the definition 
of outcomes/measures? 

® When criteria for population served is broadened to the community, how are 
the outcomes selected broadened? 

C. "Opportunity Costs" in implementing outcomes. 

• What are the costs of implementing outcomes in terms of time, resources, and 
the avoidance of other options? 

n. Implementation in the context of standardization versus flexibility. 

A. Effects on the development of programs. 

• Do communities have a responsibility to respond to standardized outcome 
goals or do those setting standards need to allow unique definitions and 
standards as a function of community variables? 

B. Effects on program evaluation. 

• Is it possible to evaluate the effectiveness of the framework when programs 
are unique? 

• Do uniquely successful community programs need to be tested against a larger 
set of standards? 

C. Criteria for implementation. 

• Do we know what the necessary and sufficient standards are to evaluate and 
critique a selected model? 

• Do we know what critical elements must be present in a community that will 
enable a model to succeed or, absent, fail? 




^8 



Opportunity Costs: 
Implementing Outcomes 



I. Development of the framework 

• Community brainstorming 

• Strategic planning 

• Preparing for/recording proceedings 

• Developing the model 

• Defining the terms 

• Choosing the baselines/data base 

• Operationalizing the definitions 

• Developing the contracts 

• Developing criteria for joint selection of population 

• "Marketing" the model among partners 

Time; 800 hours 

n. Implementation of the model 

• Testing the children (baselines) 

• Compiling/Entering baseline data 

• Community development 

• Development of training package 

• Case planning and services delivery 

• Documentation of individual client outcome plans 

• Ongoing management of outcome data base: 

4-6 hours per week per worker 



Time; 1030 hours 



May 1993 - Febmary 1994 



TOTAL TIME: 1830 hours 



El Paso County, Colorado 
Department of Social Services 





PROJECT REDIRECT 
Client Outcomes 


Goal Areas 
Safety /Permanency 


Youth lives in a safe, permanent home 


Physical Health 


Youth is a physically healthy individual 


Emotional 


Youth is an emotional healthy individual 


Social/Behavior 


Able to select behaviors that meet individual needs while 
conforming to commonly accepted societal norms. 


Family 


Positive connections with family are established and/or 
maintained whenever possible. 


Achievement 


School performance will be at expectancy level. 


Vocational 


Youth is employable or possesses age-appropriate pre-vocational 
skills. 
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Operationalizing an Outcome Management Process Holistically 

Or 

How To Add O.O.M.P.H. to Your Organization 



Go Fish. 

If you give someone a fish you feed her for a day. 

If you teach someone to fish, you feed her for a lifetime. 

As in many other fields of knowledge and practice, the most productive way to deal with 
outcomes in Child Protective Services (CPS) and Child Welfare Services (CWS) is not so much 
to supply the right answers as it is to ask the right questions. The quest to construct a framework 
which could lead to a menu of outcomes for CPS, one of the aims of this Roundtable, is rooted 
in the question, "What are the right/best/true outcomes for CPS?" Such a "laundry list" would 
be useful insofar as it could provide examples of outcomes, but two other questions should first 
be asked: "What is the process by which an agency can become focused on outcomes?" and "How 
does an agency construct the outcomes which will best enable it to fulfill its stated mission?" 

We expend much effort on trying to give fish to agencies (specifying outcomes) instead of trying 
to teach agencies to fish for themselves (defming the processes by which agencies can restructure 
themselves to become outcome-focused). The objective of this presentation is to make a 
beginning sketch of such a process and to illustrate that process with examples from the Texas 
experience in moving toward an outcome-focused CPS system. 

The title of this paper contains four words or phrases which provide keys to understanding the 
p.rocess of outcome management: 

Operationalizing: This means doing something, making a start even if the current state of 
knowledge and research is imperfect. 

Outcome Management: This means actively (re)deploying agency resources to achieve 
outcomes. 

Process: Sometimes we talk as if outcomes were themselves the outcome. It is more 
productive to think of outcomes as part of a process which calls for a conceptual re- 
orientation of our current approach to service delivery and agency focus. 

Holistically: To incorporate outcomes into an agency’s process requires a holistic 
(comprehensive) approach to agency re-engineering. 

There are three steps for an agency to take on its journey towards an outcome management 
process: 



1 . Creating an agency identity; 



2. Specifying outcomes and measures; 

3. Re-structuring the agency to support outcome-focused practice. 

Get a Name. 

There is a Sufi teaching story, based on the Middle Eastern comic folk character of Mulla 
Nasrudin, which illustrates the importance of identity; 

One day the Mulla was walking the streets of the city and forgot who he was. Frightened, 
he stepped into the nearest shop. Closing the door, he approached the shopkeeper. "Now, 
did you see me come into your shop?" asked the Mulla. "Yes," replied the shopkeeper. 
Stepping closer, the Mulla asked, "Have you ever seen me before?" The now-puzzled 
shopkeeper said, "Why, no." The Mulla, throwing up his hands and now practically in the 
shopkeeper’s face, cried, "Then how do you know it is me?" (Adapted from Shah, Idries, The 
Pleasantries of the Incredible Mulla Nasrudin, E.P. Dutton & Co., New York, 1971) 

How well does this story illustrate the plight of many CPS agencies? Unless an agency clearly 
defines its identity it is at the mercy of how others choose to define it (legislature, other agencies, 
media, boards, etc). Creating an agency identity is an essential prerequisite for developing an 
outcome management process, because it provides the foundation upon which relevant outcomes 
can be constructed and stabilized. Without this step, an agency, reacting to internal and external 
storms, has little chance for steering a considered and steady course towards any goal. 

Creating an agency identity includes defining a mission and goals, determining the agency’s place 
in the spectrum of community services, deciding on target populations, and developing a strategic 
plan. It is a process which an agency must con' ;iously and deliberatively undertake, being 
careful to involve key staff and all levels of staff in the agency’s internal and external 
communities. Besides enabling the agency to better determine and steer its own course, creating 
an agency identity empowers and equips concerned advocacy groups to advocate for the agency’s 
mission and goals. 

Usually CPS agencies ’see-saw’ between seeing themselves as focused on assuring child safety 
(Child Protective Services) and seeing themselves as focused on maximizing child well-being 
(Child Welfare Services). When we implemented risk assessment in Texas in 1992 and revised 
policy to incorporate risk concepts, we clearly chose child protective services as the primary 
contribution of CPS to the overall array of community services. What follows is an excerpt from 
our position paper on the role of CPS services in the community: 

As a tertiary service, child protective services exists to serve those families in which a 
risk of child abuse/neglect has been identified and either a) the family is unable or 
unwilling to make the environment safe for the child through utilization of community 
resources, or b) the resources needed to help the family are not available or accessible 
in the community. Child protective services does not attempt to meet all the needs of 
families and children but focuses on helping families resolve those problems causing the 
risk of abuse/neglect. The goals of the child protective services program are: 
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o To provide for the immediate safety of children at risk through controlling the risk 
conditions within the family or, when that is not possible, through placing the 
children in temporary substitute care. 

o To provide for the ongoing safety of children who are at risk by strengthening 
families so that the problems causing the risk are reduced or eliminated or, when 
that is not possible, through placing the children in permanent homes which assure 
that their best interests are attained. 

Once these goals have been achieved, child protective services intervention into the 
family ends, and the primary and secondary prevention services in the community 
assume the responsibility of assuring the child’s and family’s well-being. 

Get a Map. 

Once an agency has a clear sense of identity, it is ready to take the next step, specifying 
outcomes and outcome measures. There are three aspects' of this process to highlight here. 

1 . Ensuring consistency between outcomes and agency identity; 

2. Incorporating agency values into outcomes; 

3. Attending to outcome ’dissonance.’ 

Consistency: If an agency has chosen to focus primarily upon achieving child safety, then it 
should avoid outcomes that, for example, reflect attainment of maximum individual and social 
functioning of all family members. Instead your outcomes should reflect attainment of the 
minimum conditions necessary to assure the ongoing safety of the children at risk in the family. 
Consistency will help to ensure that an agency uses its scarce resources in the most efficient and 
effective manner, since outcomes provide the rationale for staffing, service arrays, policy, etc. 

Values: To be complete and accurate, outcomes must include key values from the agency culture, 
from the profession, and from the community. For example, just to specify "child safety" as an 
outcome is misleading. After all, one might argue that we could achieve greater child safety by 
taking measures such as permanently removing every child at risk of abuse or neglect. (Of 
course, we know that children are also abused in substitute care.) Thus, we are not after just any 
child safety, but child safety in the child’s family of origin (if at all possible). And even this is 
not completely satisfactory, for upon closer examination we find that we must incorporate values 
from the Child Welfare perspective. What we are really after is child safety, within the context 
of the child's family of origin, such that the child’s environment supports at least a minimally 
acceptable level of functioning for the child. 

Dissonance: Specifying agency values is one thing. Dealing with how these outcomes harmonize 
or conflict with expectations in the community and within the agency is another thing. The issue 
here is how an agency brings along its internal and external constituencies as it defines its identity 
and outcomes. To the extent that there is agreement, all is well, and the agency can make 
progress and marshall support for its goals. To the extent that there is conflict (dissonance), the 
agency must listen, educate, advocate, negotiate, etc. 
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Start Your Journey. 



The final step in the process is the most difficult; restructuring the agency to support outcome- 
based practice. All aspects of agency functioning must be scrutinized and re-engineered 
according to whether and how much they support achievement of the specified outcomes. There 
are five key ingredients to this process, which is now commonly called "reinventing government:" 

1 . Shared agency vision within the agency and with key constituencies without; 

2. Top-down management commitment to the vision and to the process; 

3 . Commitment of sufficient time and resources; 

4. Comprehensive plan for completing the process; 

5. Ongoing support to sustain change. 

An agency will suffer some degree of negative consequences if it does not take such a 
comprehensive approach to restructuring. When we implemented risk assessment in Texas, we 
took great pains to deal with all aspects of agency functioning which we thought would be 
affected by the change to a risk-based practice. We restructured policy, practice, forms, 
management, training, legal, contracting, community relations, etc. Unfortunately, however, we 
did not re-examine thoroughly enough our data and reporting systems. Consequently, when it 
came time to prepare our budget request for the upcoming legislative session, we found that our 
data concerning case counts, services, etc., were outdated, being based upon our old, 
maltreatment-focused system rather than upon the risk assessment system. The point is that when 
an agency re-orients its practice, it must think of everything. 

To summarize, the basic principle of outcome-based systems change could be stated as follows: 

The mission of each aspect of the agency is to employ the methods which enable the 

agency to attain its outcomes in the most effective and efficient manner possible. 

What we find then is that, ultimately, a focus on outcomes returns us full circle to a renewed 
emphasis upon process. To optimize outcomes, our processes must be effective. Our challenge 
is to construct organizational system which assist us in continually making adjustments to our 
methods and resources to maximize our achievement of desired outcomes. 

The Texas Journey 

In order to meet this challenge Texas is developing a Program Assessment process. It will be 
composed of a system to generate information on a continuously interacting set of variables 
designed to influence one another. This "Continuous Quality Improvement System" provides 
information on inputs, processes, and outcomes for use in planning and managing the program 
(including use in legislative appropriation requests). See the Continuous Quality Improvement 
System diagram below for an illustration of this process. 

(Insert diagram here) 
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CONTINUOUS QUALITY IMPROVEMENT SYSTEM 
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The system will generate information on: 



1 . Inputs, such as the number of cases to be served, the dollars and staff available, 
including the dollars available for purchased services; 

2. Processes, such as the type and duration of services applied, or the number of outputs 
achieved, e.g., the number of families preserved, the number of children adopted, etc.; 

3. Outcomes, i.e., results or client impacts, such as child safety expressed as recidivism 
rates, child functioning, family functioning, client satisfaction, etc. 

The specific outcome measures necessary to make this type system work will evolve and be 
refined over time as this interactive system is used. 

It is necessary to look at all three areas; inputs, processes, and outcomes, in order to properly 
manage the program. If, for example, processes and outputs only are used it would be 
possible to track the number and quality of services without knowing what results or impacts 
it had on the clients. And if both the processes and outcomes are tracked without looking at 
inputs, it would be easy to make a mistake in judgement. For example, if the outcomes were 
less than desired, one could easily conclude that staff were not doing a good job, when the 
real reason for the poor results might be a lack of adequate resources, not poor staff 
performance. 

Our knowledge is, and probably always will be imperfect. However, a system of interactive 
input, process, and outcome data can provide us with a significantly better opportunity to 
continuously make appropriate adustments to the methods and resources used to maximize 
achievement of our desired outcomes. 
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PRESENTATIONS OF EXPERIENCE 



CALIFORNIA 



EFFECTS OF NUMBERS OF IN-PERSON VISITS 
ON RATES OF CHILD MALTREATMENT RECURRENCE 

SECOND ANNUAL ROUNDTABLE ON OUTCOME 
MEASURES IN CHILD WELFARE SERVICES 

SAN ANTONIO, TEXAS MARCH 30- APRIL 1, 1994 

BY 

WILL JOHNSON, D.S.W 
MANAGER, PLANNING, EVALUATION, 

AND RESEARCH UNIT 

ALAMEDA COUNTY SOCIAL SERVICES AGENCY 
OAKLAND, CALIFORNIA 
(510)268-2455 

Recurrence of child maltreatment is, or should be, a primary focus of the child welfare 
system. Prior to occurrence of maltreatment state statutes typically don't allow intervention. 

.-^er occurrence of maltreatment, recurrence of maltreatment is the only form of maltreatment 
left as a target of intervention. The question arises, are efforts to prevent recurrence effective'’ 

To address this question, a prospective longitudinal outcome study of N = 303 physical abuse 
cases was undertaken in the Alameda County Social Services Agency. Physical abuse cases 
entering the family maintenance caseload of the agency between 1987 and 1989 were followed for 
two years after case opening and a determination was made as to recurrence of maltreatment 
(physical abuse, sexual abuse, or neglect). The primary independent variables in the study were 
( 1 ) the number of in-person visits with family members by child welfare workers (<= 8 visits 
versus > 8 visits. Eight was the median number of visits.), and (2) the number of dollars spent on 
contract services (primarily counseling and parent training). Risk of maltreatment recurrence at 
the time of case opening was used as a control variable. It was measured using a risk assessment 
instrument validated on cases in the same sample. Logistic regression was used to assess the 
relationship between visits, contract dollars spent, and maltreatment recurrence, controlling for 
assessed risk. 

Results and Discussion 

Figure 1 shows that the rate of maltreatment recurrence for cases with <= 8 in-person 
visits was 33%, while the recurrence rate for cases with > 8 in-person visits was \ 1 %. The results 
of the logistic regression shown in Figure 2 indicate that, controlling for risk and contract service 
dollars spent,, numbers of visits were negatively related to maltreatment recurrence (partial r = - 
. 146, p < .004). Also, controlling for numbers of visits and contract service dollars spent, risk 
was positively related to maltreatment recurrence (partial r = . 183, p < .0007), indicating that the 
measurement of risk used as a control variable had predictive and construct validity. It is clearly 
noteworthy that contract services were not related to maltreatment recurrence when risk and in- 
person visits were controlled. 



While a randomized clinical trial would be required to make a definitive demonstration of 
the etfects of visits on maltreatment recurrence, the results offer encouraging evidence of the 
beneficial effects of in-person visits. Contract services appear ineffective in this data set. 
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PERCENT OF CASES WITH RECURRENCES 



FIGURE 1 



RECURRENCES AS A FUNCTION OF VISITS 
CHI SQUARE = 9.44 P < .01 
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FIGURE 2 
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Variables in the Equation 
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PRESENTATIONS OF EXPERIENCE 



UTAH 



UTAH CUSTOMER SURVEY 

Earl Bassett, Associate Director 
Utah Division of Family Services 
May 1994 



OVERVIEW: 

In Utah, we wanted to measure the service provided by our child welfare 
workers. We thought it would be interesting and very helpful if workers had the 
benefit of knowing how their performance is perceived from the people they serve or 
work with. We understand this approach was used at Boys Town in Nebraska and 
subsequently used by Utah Youth Village in Utah who utilize the Teaching Family 
Model. Eric Bjorklund and staff at Utah Youth Village volunteered to assist us in the 
development of this model for state workers of child welfare. 

If we think of all with whom we work as our customers, we include not only 
the clientele whom we directly serve but also all others who work with us in that 
process of serving the same client. This includes parents, foster parents, courts, 
providers, etc. Our effort has been to obtain feedback from them as to how they 
perceive our performance in serving them and or the client. 

The caseworker is perceived by the customers we serve, as being in a very 
powerful position. From the view of the child, the case worker determines where and 
with whom he or she lives. For the parents, the case worker determines whether they 
keep their child. For the foster parents the case worker decides if they receive foster 
children even, in some cases, the rate of reimbursement for that child. Case worker 
decision affect providers also, as they need sufficient clientele to operate their 
programs. 

For this reason, the perceived powerful position of the caseworker, customers 
are reluctant to give honest feedback due to the fear of offending the case wcrker--and 
there is too much to lose by offending the caseworker. It is said that every one 
laughs at the jokes told by the judge. The same principle applies to case workers. 

Of course there are those who love to challenge the caseworker and don't mind 
telling him or her where to go. I worked on the front line for many years, and I've 
been told where to go plenty of times. Generally speaking, however, customers are 
reluctant to offend the person who holds so much power over their lives. For this 
reason it is difficult to get honest feedback about how the work Is being done. 

Keeping this in mind, we chose to administer an anonymous survey. Customers 
were sent survey forms and asked to fill them out regarding a specific worker. Each 
form had the workers name at the top, a series of questions about the workers 
performance, and If the customer had specific questions he or she wanted to discuss. 
(See attached questionnaire) The names of the client are not included on the forms 
and respondents were assured complete confidentiality as to their response on the 
cover letter. The forms were returned in a self-addressed envelope. 
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Utah Customer Survey 
Page 2 



As you might suspect, our return rate was not as we had hoped and we found 
it necessary to remind people to return the form. Calls were made to the customers 
asking that they complete and return the survey. This effort increased our responses 
to over 50%. We also sent reminder postcards about the questionnaire. The 
postcards produced little results. 

Data from questionnaires were then entered on the computer using a DataBase 
software database. Reports, both in the graph format and numerical format, were 
printed with the data by worker. (See copy of attached reports.) 

The real strength of this approach is to obtain data by worker as opposed to 
feedback by agency. There is a believe that we judge others by their behavior and 
ourselves by our intentions. Since we always have good intentions for what we do, 
we allow ourselves to believe that someone else is making the mistakes. Our 
approach gives workers feedback that relates specifically to them. Incidentally, our 
workers received a lot of positive feedback as well which validated their efforts and 
motivated them to continue their hard work. 

The reports were then shared with the worker, his or her supervisor, and the 
assistant director or director. Each supervisor was trained to utilize the data to assist 
the worker. 

PROBLEMS: 



As you might suspect, we ran into a variety of problems. 

1 . Some workers were reluctant to participate, so we asked for volunteers on the 
first pilot project of the survey. Some workers were anxious for the feedback 
and some were afraid of it. 

2. Secretaries viewed this as extra work and resisted helping. The sending of 
different questionnaires to various people was confusing for the harried 
secretaries and a couple times the wrong questioners were sent. 

3. Some providers didn't know the workers and did not give the questionnaire to 
their staff who did. 

4. Some providers received many questionnaires for several staff members and 
therefore, resisted the effort to complete the forms. 

5. Some customers expressed felt the confidentiality would be breached. 

6. One juvenile court judge felt it was inappropriate for him to complete the forms. 
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Utah Customer Survey 
Page 3 



FUTURE CHALLENGES: 



1 . Manpower: We are solving the man power problem by converting most of the 
tasks to be done to computer. We will down load the customers names and 
address labels from the computer. Letters will be stuffed by mailing machines. 
The questionnaires are now being reformatted so they can be read by a scanner 
and the data electronically transferred to the region who can run the reports. 

2. Baseline data: Once we get the data rolling we will need to develop base line 
data to determine what the tolerance level will be from the Norm. It will take 
some time to complete this part. 

3. Costs: So far we are having to eat the costs from our existing budget. This 
does not make the program popular. 

4. Validity: Validity of the survey needs to established. 

Despite the problems and challenges ahead of us, I believe the product will be 
worth the effort. Once workers realize they are being measured by others outside the 
agency, such knowledge should promote an improvement in their efforts and behavior. 
The positive feedback is probably worth all the effort. Another great benefit comes 
when complaints are received regarding a certain worker. Survey results can give 
additional data to defend a worker or to help a worker make a change in how they do 
business. 



FOSTER PARENT FEEDBACK 



Caseworker Caseworker Name ~ Date June 3i ^993 

PIe 2 S 6 answer each question by marking the ONE circle that most closely reflects your 
feelings as they pertain to the caseworker listed above. 





■\LMOCT 

VLWAVS 


USUALLY 


RARELY 


NEVER 


NO 

PERSONAL 

knowledge 


i. This caseworker is courteous 




o 


o 


o 


o 


2, This caseworker involves you in 
treatment decisions concerning the 
child(ren) 




o 


o 


o 


o 


3. This caseworker is professional in 
their dealings with you 


' ' 


o 


o 


o 


o 


4. You received a placement 
information form (Form 952) from this 
caseworker 




o 


o 


o 


o 


5. You are satisfied with the amount 
and quality of in-person and telephone 
contacts this caseworker has with the 
child(ren) 


0 


o 


o 


o 


o 


Average number of contacts per month 

6. You are satisfied with the frequency 
and quality of communication you have 
with this caseworker 


0 


o 


o 


o 


o 


Average number of contacts per month 

7. You are satisfied that the caseworker 
is supportive of the iong-ienn and short- 
term goals for the child as outlined by 
the treatment plan 


0 


o 


o 


o 


o 


8. You arc satisfied that the caseworker 
advocates for the best interest of the 
child 


0 


o 


o 


o 


o 


9. This caseworker is appropriate in 
their dress and conduct 


0 


o 


o 


o 


o 



10. Any additional concerns, comments, or suggestions concerning this caseworker? 



If you htvt a problem with thia cauwof ker, would you like an admmiiuntof to call you? If ao, pieaac put your name and phone number below. 
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Name 
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Phone number 






YOUTH QUESTIONNAIRE 



Caseworker Caseworker name - 



Date June 3. 1993 



Please answer each question by marking the ONE circle that most closely 
reflects your feelings as they pertain to the caseworker listed above. 



1 
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ALMOST 

always 


USUALLY 


RARELY 


NEVER 


NO 

PERSONAL 

knowledge 


1 . Your caseworker is helpful 


0 


o 


o 


o 


o 


2. Your caseworker is concerned about 
you and your success in the future 


0 


o 


0 


o 


o 


3. Your caseworker is courteous 


0 


o 


o 


•0 


o 


4. Your caseworker supports you in 
achieving your treatment plan goals 


0 


o 


o 


o 


o 


5. You are satisfied with the number of 
phone contacts with your caseworker 


0 


o 


o 


o 


o 


6. You arc satisfied with the number of 
visits with your caseworker 


0 


o 


o 


o 


o 



7. Your caseworker has discussed your treatment plan with you Yes O No O 

8. Your caseworker involved you in ibe seieciion of your current placement if applicable 

9. Any additional concerns, comments, or suggestions concerning your caseworker 



If you have a problem with this caseworker, would you like admimsirttor to call you? If so, please pul your 
name and phone number below. 



Name 



Phone number 
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PROVIDER FEEDBACK 



Caseworker Caseworker Name- 



Date -Tune 3. 1993 



Please answer each question by marking the ONE circle that most closely reflects your 
feelings as they peitain to the caseworker listed above. 





ALMOST 

ALWAYS 


USUALLY 


RARELY 


NEVER 


NO 

PERSONAL 

KNOWLEDGE 


1. This caseworker is courteous 


0 


o 


0 


o 


o 


2. This caseworker involves you in 
treatment decisions concerning the child 


o 


O 


0 


0 


o 


3. This caseworker is professional in 
their dealings with you 


o 


O 


0 


o 


o 


4. You received placement information 
forms (Form 952) from this caseworker 


0 


O 


0 


o 


o 


5. You arc satisfied with the amount 
and quality of in-person and telephone 
contact the caseworker has with the 
child 


o 


o 


0 


o 


o 


Average number of contacts per month 












6. You arc satisfied with the frequency 
and quality of communication you have 
with the caseworker 


o 


o 


0 


o 


o 


Average number of contacts per month 












7. You arc satisfied that the caseworker 
is supportive of the long-term and short- 
term goals for the child as outlined by 
the treatment plan 


o 


o 


o 


o 


o 


8. You arc satisfied that the caseworker 
advocates for the best interest of the 
child 


0 


o 


o 


o 


o 


9. This caseworker is appropriate in 
their dress and conduct 


o 


o 


o 


o 


o 



10. Any additional concsras, comments, or suggestions concerning this caseworker? 



If you have a problem with this caseworker, would you like an administrator to call you? If so, please put your 
name and phone number below. 
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Name 
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question # 
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Sene = 5 
Responded = 4 




DATE . Jan 93 

YOUTH CUESTICNNAIRE 


QUESTION » 


ALMOST 
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NO 

PERSONAL 

USUALLY RARELY NEVER CNOWLEDGE 






■3 “1 0 0 0 
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0 

2 

L 



16 
' = i 



6 

25% 



0 

0 

0 

0 

0 



0 

0 % 



0 

0 

0 

0 

0 



0 

0 % 



0 

0 

0 

0 

0 



0 

0 % 



7 

8 
9 



YES 



NO 



0 

1 



SHE PUSHES ME TO STAY ON THE RIGHT TRACK. 
SHE'S A GREAT PERSON AND WE GET ALONG GREAT. 
I LOVE HER. 
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Questionnaire Summary 




Questionnaire Title 



Foster Parent baxistaction 





Worker Characteristics 



WORKER NAME 
Provider Satisfaction 




Worker Characteristics jppy ftVAllABlE 




RECOMMENDED OUTCOMES FOR 
RESIDENTIAL CARE 



Eric W. Bjorklund, J.D. 
Utah Youth Village 



INTRODUCTION 

One of the most expensive 
segments of the child welfare system 
Is residential care. Caseworkers place 
children with residential providers for 
many different reasons. Residential 
providers include basic foster care, 
treatment foster homes, group homes, 
residential treatment centers, secure 
facilities, etc. 

Traditionally, a caseworker 
seeks certain activities which are 
referred to as "process" components 
when placing a child with a provider. 
Process components are activities or 
services provided for the child. For 
example, the caseworker seeks a 
place for the child to live. The child 
will be provided a certain amount of 
individual therapy, group therapy or 
skills training. A tracker may be 
assigned to the youth. 

it is assumed that the 
"process" components produce the 
desired outcome of a functional child 
who successfully leaves the child 
welfare system. 

Unfortunately, we do not really 
know what correlation certain 
"process" components of treatment 
have with outcome results. This is 
complicated further by the varying 
competence of different providers. 



Outcome measures help us 
determine if we are meeting our goals. 
With regard to residential care, the 
goal of the child welfare system 
should be to move children from 
residential care to productive 
independent lives in society. 

In theory, a caseworker's 
placement decision should be driven 
by this goal. The caseworker should 
ask herself, "Will this placement 
ultimately move this child from 
residential care and cause him to be 
functional in the real world?" 

CURRENT BOTTOM-LINE OUTCOME 
MEASURES 

In practice however, the 
"outcome measurements" fostered by 
the everyday dynamics of the system 
are varied and sometimes 
questionable. When a caseworker 
wonders where to place a child he/she 
may ask: 

1 . Which provider controls or 
hides the youth's behavior so 
that I am not constantly dealing 
with issues? 

This is a combination of the 
pressure of excessive caseloads 
and a common belief that no 
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news is good news, in reality 
some ongoing contact with the 
caseworker may or may not be 
positively related to creating a 
functional youth. 

2. Which provider will 
immediately and unconditionally 
take a youth that needs to be 
placed? 

Ironically, this may be 
successfully completed before 
treatment has actually begun. 

It can also equate to a license 
to warehouse a child. 

3. Which provider will help this 
child "work through the child's 
issues?" 

This may exalt certain 
treatment processes without 
regard to the provider's ability 
to produce a more functional 
child. It may be a more 
accurate predictor of a 
therapist's employment than 
the child's behavioral change. 
Moreover, this "outcome" is so 
defuse and nebulous that its 
achievement is unmeasurable. 

4. Which treatment provider 
most closely aligns with the 
theories of my college 
professors or the latest child 
care fad? 



treatment or that the 
professor's theories may have 
been wrong. 

5. Which provider employs 
someone I know and trust? 

The caseworker may 
unfortunately know and trust 
the least competent provider. 

Yet how can he/she be blamed 
if administrators do not provide 
better data and guidance about 
who is achieving success. 

6. Which provider has done 
well with the children I refer to 
them? 

All too often the 
caseworker's experience may 
be limited. Moreover, his/her 
criteria of prior success may be 
listed in one of the immediately 
preceding paragraphs. 

7. Which provider will provide 
as many resources as possible 
to the child in the least 
restrictive setting? 

This is appealing to a 
caseworker. If the caseworker 
involves as many people as 
possible in the child's care then 
the caseworker is beyond 
criticism and feels he/she has 
done the very best for the child. 



This does not consider that a Yet, this provides a classic 

certain provider may not example of using "process 

effectively apply a claimed components" instead of 

"outcome measures." For 
example, it is our experience 



that highly skilled foster 
couples will predictively achieve 
better outcomes, by 
themselves, than will a nominal 
foster couple combined with a 
tracker, an individual therapist 
and a group therapist. 

A child prefers to interact 
with a limited number of adults. 
The fewer number of adults 
increases the likelihood of 
bonding and decreases the 
child's ability to manipulate the 
adults against each other. 

More chefs in the kitchen do 
not necessarily yield a better 
soup. However, the truth will 
be known only when the 
system starts to routinely 
measure outcomes. 

I recently talked with the 
director of an agency which 
provides much of the tracking 
services to the state of Utah. 
She acknowledged that trackers 
were probably a useless 
service. She doubted that the 
addition of a tracker to a child's 
treatment plan would be 
predictive of outcome success. 
Yet, the caseworkers demand it 
because of the deceptive 
comfort it offers them. The 
provider responds accordingly. 
More money is expended. 

Actual behavior may not be 
changed. 

CASEWORKERS DRIVE THE RESULTS 

Providers quickly learn which 

processes or "outcomes" drive 



placement decisions in the current 
child welfare system. To do 
otherwise is to insure empty beds and 
financial despair as a provider. 

Provider's services are usually 
process driven. Based on the 
demands from government agencies. 
Unfortunately ultimate outcomes are 
usually irrelevant. 

It is as if a business hired 
numerous sales people, payed them 
millions of dollars, but never evaluated 
their sales. How efficient would this 
business be? 

It is the caseworker's headset 
which drives the whole system. 

Sadly, caseworkers are left adrift in 
the seas of decision-making by 
administrators who will not articulate 
legitimate outcomes, will not measure 
those outcomes and will not 
communicate the outcomes. 

It is the administrators who 
must empower caseworkers to make 
placement and assessment decisions 
which predict valid outcomes. 

inFAl OUTCOME CHARACTERISTICS 

Faced with this challenge, we 
suggest outcome measures with the 
following characteristics: 

1 . The outcome measures 
would motivate the providers to 
change the child's behavior so 
that the child would leave 
residential care successfully. 



Children who are moving 
into residential treatment have 
difficulty functioning In 
traditional family structures and 
in the public school system. 

For these children to 
successfully leave residential 
care they must become 
functional in family 
environments and in the school 
system. 

2. The outcome measures 
need to be essentially fair to 
the providers. Judgements on 
outcomes would have to take 
into account the comparative 
difficulty of youth. Providers 
would need to be compared 
with others who deal with 
comparable youth. 

3. Measures should be as 
objective and independent as 
possible. 

4. The outcome data would 
need to be of such a nature 
that it can be periodically 
published to caseworkers and 
providers. Without publishing 
the data, providers may not 
change their behavior and 
caseworkers may not change 
their attitudes. 

5. The outcome data must not 
be too expensive or 
burdensome to obtain, 
especially considering the goal 
of periodic ongoing publication 
of the data. 



TWO GENERAL OUTCOME 
MEASURES 

We advocate two general 
outcome measures: movement to less 
restrictive environments and school 
performance. The data to measure 
either outcome is already in most 
governmental systems and 
comparitively inexpensive to collect. 

OUTCOME #1: 

Movement to a less restrictive 
environment when a child's placement 
with the provider facility is terminated 
is the first outcome measure. For 
example, when a child leaves a group 
home to go to foster care, 
independent living or her/his family, it 
would be counted as a success. 

When the same child runs away, goes 
to another equally or more restrictive 
setting it would be counted as failure. 

This outcome measure is 
intended to encourage a provider to 
teach a child to succeed in less 
restrictive family-like environments. 
This outcome measure focuses on the 
youth's development of family 
competence and the youth's 
movement out of the system. 

This outcome measure is 
tracked as a percentage and would be 
reported as an ongoing cumulative 
score, much like a batting average. 

Variations on this outcome 
measure would include determining if 
the child maintains the less restrictive 



status for some duration of time after 
termination from the program. 

One might further look at how 
stable (movement from placement to 
placement) the child is for some 
duration of time after termination. 
Again, this data is already in the 
system and comparatively easy to 
access. 

However, one must be careful, 
because the ability of the successor 
provider is also being measured when 
measuring youth status for some time 
after termination from a program. A 
youth's maintenance in a less 
restrictive environment and relative 
stability may be more a function of 
the successor foster home's 
competence than the treatment ability 
of the prior group home. Yet with a 
sufficient number of youth, statistical 
trenc-s should emerge to suggest if a 
provider is doing well or may need 
help. 

When measured by this type of 
outcome, providers are motivated to 
use, find, and create treatment 
methods that achieve movement to 
less restrictive environments. This 
might include a greater emphasis on 
working with the target family, a 
different approach to psychotherapy, 
the introduction of skills building 
treatment programs, etc. 

OUTCOME #2: 

School performance, the 
second outcome measure, is assessed 
through grades, citizenship and 
attendance of the child at school. A 



child's performance in school is 
widely recognized as a gross indicator 
of his/her functionality. Moreover, a 
child's failure in the school system is 
usually a predictor or contributing 
cause of their placement in the child 
welfare system. It is therefore 
important to use this outcome 
measure to motivate providers to 
teach children to succeed in the 
school system. 

Numeric values would be 
assigned to each criteria (grades, 
citizenship and attendance) to yield an 
overall numeric score. This outcome 
measure would be reported by the 
ongoing average score of all of the 
children in the facility. This data is 
readily available for most children in 
the system. Depending on your 
agency structure the data can be 
obtained from the school system or be 
provided through the providers. 

An important variation on this 
outcome measure is the improvement 
or regression of a child's school 
performance compared to their prior 
placement. This Is perhaps more 
indicative of the provider's 
competence. Moreover, it alleviates 
the concern that providers would 
become selective about referrals to 
get children who are more likely to 
have high grades -- e practice referred 
to as "creaming." 

DISPURSMENT 

We recommend publishing the 
running totals of the data on a 
quarterly basis to all caseworkers and 
all providers. No mandatory 



consequences should be attached to 
providers performance. The outcome 
measures are not accurate enough to 
justify mandatory consequences. The 
publication of the data should be its 
own motivator. 

GENERAL CONSIDERATIONS 

These outcome measures 
create data which are the result of 
objective events. For example, the 
movement of a child at his placement 
termination is an independent event 
which cannot be skewed by the 
provider. Similarly, the child's school 
performance is usually provided by 
independent school teachers. To this 
extent the data is more objective than 
self-serving outcomes that could be 
manipulated by providers. 

The model or method of 
achieving success on these outcome 
measures should be irrelevant to 
government agencies, to the extent 
the provider uses ethical methodology 
and complies with the law. These 
types of outcome measures can 
motivate a provider to radically 
change its treatment system to obtain 
better outcomes. We must become 
more focused on achieving the goals 
and not just implementing the means. 

These outcome measures may 
also encourage providers to interact 
more with each other. Providers who 
show poorly on these outcomes may 
quickly seek out those who show 
strongly to solicit their training. 

It is important to track the data 
based on a facility and not on a 



provider. In our state's system, one 
provider operated many treatment 
foster homes. Children were shifted 
often from one home to another. 
Because our state's documentation is 
set up on a provider basis, that 
movement was invisible to the 
government. Its impact on the 
children, however, was likely 
devastating. When outcome data is 
reported by a facility, abuses of the 
system can be quickly detected and 
stopped. 

The outcome data may be just 
as easy to get for foster homes as for 
more restrictive care. We urge foster 
homes be included in this outcome 
process. We have all seen foster 
homes that are terrible but are 
nevertheless used often because they 
are willing to ^ake any child at any 
time of the day or night. 

We have also seen very 
competent foster parents to whom 
youth are not referred because the 
foster parents are not well known by 
caseworkers. This type of data could 
cull out incompetent foster care, while 
encouraging the use of the best foster 
parents. 

Public, as well as private 
providers, should be subject to the 
outcome measures. Providing the 
best outcome results for children must 
be the goal of a government agency, 
not protecting its own programs. 

We are concerned about 
minimizing comparisons between 
providers who deal with different 
populations - the proverbial "oranges" 
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vs. "apples" problem. When data are 
published it is important that basic 
foster homes be reported as a group. 
Each level of residential treatment 
should be compared to itself. 

We suggest segregating the 
providers by pay rate. Theoretically, 
the programs receiving the same pay, 
per youth per day, are helping children 
whose difficulty is roughly equivalent. 
We would suggest down-playing 
claims of "oranges" vs. "apples" 
within the same pay strata. A 
provider will often believe that the 
children it serves are the most difficult 
in the system. Ironically, it usually 
knows little or nothing about the other 
providers' children. 

We have had concern that 
measuring performance will cause 
providers to "cream" the youth to 
obtain those who will yield the desired 
outcome. It may not be bad, 
however, if a provider discovers that 
they cannot positively impact children 
of a certain difficulty level. 

The measuring of performance 
will likely cause some providers to 
reassess who they are equipped to 
deal with successfully or to change 
some of their treatment 
methodologies. 

Nevertheless, the provider who 
tries to screen for the easy children 
will find their referral base dwindling 
and other, more capable,, providers 
taking their place In the system. The 
financial motivation of providers to 
keep beds full outweighs the 
temptation to skim for the easy 



children. 

It must be acknowledged that 
these outcome measures are not 
empirical. They provide gross 
indicators of competence. It is for 
that reason that no mandatory level of 
performance is attached to the 
system. Simply the reporting of the 
data provides its own motivation. 

THE ANTICIPATED RESULTS 

The anticipated impact of these 
output measures will be two fold: 

1 . Caseworkers will start to 
think in terms of "outcomes" in 
addition to "process." Administrators 
will finally articulated simple outcome 
goals for the residential care system. 
Caseworkers will have data to guide 
the placement of children. Children 
will be less likely to be abused by the 
system. The system will also save 
money due to increased efficiencies. 

2. Providers as a group will 
pay more attention to teaching 
children to be competent in family 
environments and in schools. Even if 
the caseworkers never saw the 
outcome results, the publication of 
the results amongst providers would 
increase the quality of care given to 
the children. Simply measuring a 
behavior, may change the behavior. 

It is our belief that over time an 
agency will find that most of the 
providers will rate similarly to each 
other. A few exceptions may show 
very poorly. Those may be the 
providers who are warehousing 
children. They will become motivated 



to improve their treatment system or 
their referrals will rightfully diminish. 

CONCLUSION 

The ultimate question becomes, 
"Is it better to measure outcome 
performance using gross but 
economical measures or to avoid it 
because of the potential for unfair 
results?" We suggest that the 
benefits of measuring competence far 
outweigh the risks. We would see 
the trade-off to be similar to the 
question, "Is it better to drive 
automobiles or not drive because of 
the potential risk for an accident?" 

We must start to measure 
competence, albeit, imperfectly. 

One additional intriguing use of 
this data might be to sort it by 
caseworker as well as by provider. 

This may quickly highlight those 
caseworkers who are routinely 
referring youth to ineffective providers 
for whatever reason. 

It may also lead to the 
discovery of other activities that are 
common to successful caseworkers. 

It might motivate caseworkers to seek 
the treatment that will provide the 
best outcomes for youth. Note that 
this is a different goal that seeking the 
best treatment for youth. 

However, we would be careful 
of the timing of the implemention this 
second step. Caseworkers may 
overreact to the outcome measures 
added to the system, let alone 
focusing the measures on them. 
Overreaction may have unfair 



implications for certain providers. We 
would suggest the provider data be 
given substantial time to become 
routine before the second shoe fail. 

Occasionally, more empirical 
studies looking at many other 
outcome measures could be compared 
to the data generated by these two 
outcome measures. This could be 
used to assess the validity of the 
proposed measures and better 
understand how they relate to other 
data. 
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WORK GROUP SESSIONS 



APRIL 8, 1994 



WORK GROUP SESSIONS 



On the second day of the Roundtable, participants divided into four separate groups 
that were defined around one of four target outcomes: 

• child safety, 

• family continuity/family preservation, 

• child functioning, and 

• family functioning. 



For the target outcome, each group was asked to: 

• clarify the definition of the target outcome; 

• review illustrative examples of indicators and measures; and 

• generate a list of additional indicators. 

At the end of the day, the participants reconvened so that each work group could 
report back and summarize their work to the larger group. Time was allotted for 
discussion and reaction to the list of additional outcome indicators developed by each 
work group. Each group's list of outcome indicators were word processed and copied 
overnight and made available to each participant the following morning. 

Based on the previous days feedback from the larger group, the work groups 
prioritized their list of outcome indicators and identified no more than five outcome 
indicators for each target category. Each group, using sheets of transparencies and 
markers, recorded their priority list of outcome indicators, and again presented them 
to the larger group for discussion and reaction. 

The following pages represent the work group's priority draft lists of indicators 
generated during the sessions. In order to reflect the original intent of each work 
group and avoid losing any subtleties of meaning, AHA minimally edited and revised 
the indicators. 
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Second Annual Roundtable on Outcome Measures 
in Child Welfare Services 

Work Group Results > Draft Indicators 
Friday April 8, 1 994 



Child Safety Work Group Notes 

Child Focus 

1 . Decrease the factors that put adolescents at-risk of perpetrating violence or 
becoming a victim of violence. 

2. Decrease the number of subsequent confirmed reports of unsubstantiated cases 
of abuse or neglect. 

3. Decrease the severity and number of incidents of subsequent abuse and neglect 
within 1 2 months of initial substantiated report. 

4. Decrease the number of preventable Injuries/deaths resulting from abuse or 
neglect. 

5. Decrease the number or severity of confirmed abuse/neglect for children In out- 
of-home care. 

Parent/Famiiy 

1. Reduce domestic violence in families with children receiving child welfare 
services. 

2. Reduce subsequent maltreatment of siblings or subsequent births. 

3. Increase the use of appropriate discipline by parents with their children. 

4. Improve parents' knowledge level of the physical and developmental needs of 
children. 

5. Decrease unnecessary disruption of families as a result of the investigation of 
reports. 

Community 

1 . Increase the number, quality, and the response to reports of suspected child 
maltreatment in compliance with the law. 

2. Increase the availability, access, and adequacy of an array of services for 
children and families who have been reported/confirmed for child maltreatment. 

3. Reduce rate of child maltreatment by caretakers in positions of trust such as 
day care providers, scout leaders etc. 

4. Increase awareness of child protective services' role, in partnership with the 
community, to ensure child safety. 
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Work Group Results - Draft indicators 
Friday April 8, 1994 

Family Continuity/Preservation Work Group Notes 



This work group developed the following definition for family continuity/preservation: 

"To ensure that whenever possible children and families, however defined, remain 

safely together, are reunited as quickly as possible, and maintain kinship ties. 

Child 

1 . Improve the stability of children's living arrangements by maintaining sibling 
groups whenever possible and appropriate, and by increasing the number of 
local community placements for children in out-of-home care. 

2. Decrease the number of status offenders and runaways. 

3. Decrease the over-representation of children of color in out-of-home care. 

4. Increase the number of open adoptions, working with children to determine 
permanency goals. 

5. Improve children's feelings of ethnic, cultural and racial identity and self-worth. 

Parent/Family 

1 . Increase the number and the frequency of visits that occur between parents 
and other family members, and children who are in out-of-home care. 

2. improve the quality of parent/chiid interaction and relationship satisfaction for 
children reunited with their families. 

3. Increase contact between the permanent family and the out-of-home care 
provider(s). 

4. Increase the number of families who recognize children as a daily priority, and 
whose actions to children reflect community standards. 

5. Increase the number of families who seek supportive services and community 
resources to support the family prior to a crisis. 
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Community 

1. Increase the number of adoptive and foster care homes that are culturally 
competent, and that reflect the cultural and ethnic makeup of children in out-of- 
home care. 

2. Improve formal and informal supports for kinship care as a child welfare service. 

3. Increase community's education and outreach support to sustain and support 
families, family continuity, and family preservation. (Note: The work group 
recognized a growing tension between family preservation and child safety.) 

4. Increase community resources to help resolve parent/child conflict. 

5. Increase the number of families with basic economic supports. 



The work group recognized 'that most of these indicators, more programmatic in 
nature, fit into the other three categories. In developing these outcome indicators, the 
work group interpreted the term "family preservation" not as a specific program 
model, but more as support services to families. The group dialogued about the over- 
representation of children of color in the child welfare system, with concern that many 
are losing their identity. The group's intent was to address the need for each 
community to provide culturally and ethnically competent foster parents for children 
in out-of-home care. 
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Work Group Results > Draft Indicators 
Friday April 8, 1 994 

Ch‘‘ ' Functioning Work Group Notes 



Child 

This work group approached defining outcomes in "domains" rather than delineating 
a specific set of indicators for each domain. The general outcome domains would 
then include a specific set of differential standards based on the child's age, whether 
the custodian of the child is the natural parent or the state, etc. The five domains are: 
1) physical; 2) behavioral; 3) emotional; 4) cognitive; and 5) social. 

Parent/Family 

1. Improve application of parenting skills appropriate to child's level of 
development. 

2. Improve parents ability to access and use formal and informal community 
resources. 

3. Improve parents ability to identify child's needs. 

4. Improve parents and family members awareness of how their behavior impacts 
the child. 

5. Improve parent/child interaction. 

Community 

1 . Increase the community's willingness to become involved with dysfunctional 
families. 

2. Improve the community's involvement in improving/supporting child functioning 
within the five previously defined child domains. 

The work group commented that specific outcomes should also be defined around 
specific issues. For example, under "Community" there should be specific language 
around institutions such as schools. In recognition of limited time, the work group 
also noted that the defined outcome indicators did not address collaboration. The 
group also noted that the difference between the availability and use of [community] 
supports, suggested the need to define another set of indicators regarding utilization. 



Work Group Results - Draft Indicators 
Friday April 8, 1994 

Family Functioning Work Group Notes 

Child 

1 . Reduce child's fear of family members. 

2. Improve child's understanding of family rules. 

3. Improve child's communication techniques and negotiation skills with family 
members. 

4. Reduce family violence. 

5. Improve quality of attachment between child and family. 

Parent/Family: 

1 . Improve family's communication to effectively resolve family conflict and to 
decrease parenting/family stress level. 

2. Increase family's display of appropriate role behaviors (i.e., decreasing 
prevalence of familial substance abuse), ^nd the promotion of important values 
(such as education). 

3. Increase family's ability to identify and access community resources to resolve 
problems, and to establish and use internal and external support networks. 

4. Improve family's ability to provide continuity, caregiving, nurturing and caring. 

5. Improve parent's capability to provide basic care (i.e., nutrition, hygiene, 
supervision) for children. 



Community 

1 . Increasa the number of community-based foster parents and adoptive parents 
as long term supports. 

2. Increase availability and family use of such local support services as respite 
care, child day care, mentoring, and tutoring services. 
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3. Support local community councils to monitor and report on quality of family 
life. 

4. Create more opportunities for the child welfare system to partner with 
neighborhood resources (recreation and churches), and in community 
development efforts. 

5. Expand the number and quality of available family development services. 



GETTING STARTED 



ERLC 



97 



Some Measurement Issues to Consider 
When Developing Outcome Measures for Child Welfare Services 



Raymond S. Kirk, Ph D. 

Associate Director, Human Services Research and Design Laboratory 
University of North Carolina at Chapel Hill 



At the First Outcome Measures Roundtable, I was asked to comment on what we, 
as a profession, had done so far to measure outcomes, and what we had learned in the 
process. During my remarks I suggested that we have “measured the wrong things, and 
that we have measured them poorly.” This comment was in reference to our history of 
measuring process variables or agency activities, and making often unwarranted leaps of 
faith that these activities are somehow associated with improved outcomes for children or 
families. Rarely do we actually measure outcomes directly. 

I also suggested that as we should shift our measurement practices to focus more 
appropriately on outcome measures, and that we develop a range of outcome measures 
that can inform practice as well as program and policy. A danger in focusing on a small 
number of outcome measures (or in a worst case, a single outcome measure) is that we 
will sometimes fail to notice meaningful, positive changes in child or family conditions that 
fall short of the overall outcome. If we do not account for these smaller changes, we may 
fail to build upon intermediate successes, and commit the error of assuming that the whole 
program or intervention is a failure. 

There is increasing agreement among caseworkers, researchers and program 
directors that we nave been “measuring the wrong things,” if we want to know the results 
that our services are producing for families and children. However, that does not mean 
we should stop measuring process variables or agency activities. It is legitimate to ask 
questions about caseload sizes, numbers of investigations completed, numbers of units of 
services delivered, and so on. These numbers can provide answers to efficiency questions, 
they can inform quality assurance reviews, and they can be linked directly to resource 
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allocation decisions. However, they often tell us little about effectiveness because they fail 
to answer the effectiveness question; are the families and children better off as a result of 
these activities. Decisions about what to measure should be driven by the salient research 
or evaluation question. 

This Second Outcome Measures Roundtable has been structured to help us 
identify and agree on a set of outcome measures appropriate for child welfare. This is a 
good thing, since we have seemingly agreed to “measure the right things.” However, we 
should keep in mind that obtaining outcome measures may pose new challenges. After all, 
we are very practiced in measuring process and activity, but we are neophytes when it 
comes to measuring outcomes. Having decided to “measure the right things,” assuming 
we are interested in assessing effectiveness, we should address some of the measurement 
issues affecting our ability to determine child and family outcomes. 

Fraser (1990) recently suggested that both “proximal and distal” measures are 
needed when evaluating family preservation services. While his frame of reference was 
limited to family preservation, the reconunendation for multiple outcome measures is 
certmnly generalizable to the broad array of child welfare services. I certainly agree with 
Fraser’s (1990) suggestion that outcome measures not be thought of only as an end-point. 
There are multiple opportunities to assess outcomes during a typical child welfare 
intervention. There are also several strategies for obtaining good data when assessing 
outcomes. I will develop these assertions more fully with reference primarily to family 
preservation services, since that is the area in which I am working presently, but the points 
I will make are largely generalizable to other intervention strategies. 

The assertion that outcome measures should be assessed throughout the “life” of 
any interventioa (and thereafter, whenever possible) is predicated on the notion that we 
need to be informed about meaningful changes even if they fall short of some ultimate 
outcome measure. One of the best examples of this requirement that I know of is the 
study of Minnesota’s Family Preservation Services program conducted by AuClaire and 
Schwartz (1986). In this study they did not find significant differences in the placement 
rates of children in families receiving family reservation services (FPS) when compared to 
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presumably similar families that did not receive those services. However, if placement did 
occur, significant differences were reported in the number of placement days utilized. 
Children fi’om families that received family preservation services experienced significantly 
fewer placement days than did cWldren from families who had not received those services. 

AuClaire and Schwartz (1986) also suggested, based on f reliminary data, that in 
addition to the shorter duration of placement, families that received family preservation 
services required less intensive follow-up services than did families that did not receive 
those services. 

Had they used the “traditional,” ultimate outcome measure of placement 
prevention as the sole evaluation statistic of family preservation services, the authors 
might have concluded that the Minnesota FPS program was not successful, based on the 
finding of “no difference” between the control and experimental groups in the study. 
However, the findings of “fewer placement days” and “need for less intensive follow-up 
services” for FPS-receiving families must surely fall on the “success” side of the 
effertiveness argument. Had these intermediate outcome measures not been collected 
along the way, practice might not have been informed about some other signs of success 
for the overall program from which program modifications might be made. (In fact, those 
positive intermediate outcomes beg some other interesting questions about the premature 
use of experimental research designs in the evaluation of new programs and interventions.) 

We have seen that gathering intermediate outcome data is no less important than 
gathering outcoine data at the time of case closure or service termination, but it is also 
important to gather selected outcome data periodically after case closure or service 
termination. One benefit of post-service outcome measurement is that it provides 
information about the durability of the intervention. Again, however, multiple measures 
are desirable. Some outcome measures are imbued with measurement issues that can 
provide information that is misleading. This is true of some process measures as well, as I 
have suggested with regard to the national system for tracking foster care utilization (Kirk, 
1993). 
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The “placement prevention” statistic for family preservation is one such outcome 
measure. We have already determined that it is inadequate as. a stand-alone measure of 
program success at the time of case closure, but consider what happens to the 
“performance curve” of this measure as time passes after case closure. With the passage 
of time, some families that received family preservation services will experience the out- 
of-home placement of a child. This placement decision may be necessitated for reasons of 
child safety that were not resolved during the FPS intervention. On the other hand, the 
placement may be necessitated by factors complete unrelated to the original family 
preservation services case. Either way, the placement will probably be interpreted as a 
“failure” of the FPS intervention. This interpretation is likely because the very nature of 
the measure predisposes negative interpretations of gathered data. Specifically, since 
“placement prevention” can be characterized as the “absence of a negative outcome”, any 
time placement occurs it will fall into the “failure” category of outcomes for family 
preservation. The future curve of the data can never improve, it can only deteriorate. The 
apparent deterioration may be due to the measurement issues inherent in the statistic. At 
the very least, we should simultaneously gather potentially countervailing, “positive” 
outcome measures (such as occurrences of behavior indicating improved family 
functioning) to inform our interpretation of the apparent deterioration in the placement 
prevention statistic. 

In addition to employing measures that are both proximal and distal to the 
intervention, Fraser (1990) suggests that the measurement system should include both 
client- and program-level indicators. We have currently employed this strategy in North 
Carolina, where the state law defining and enabling family preservation services requires 
that families be contacted one, two and three years after service to find out whether the 
family is intact. Given the measurement issues associated with this mandate and the 
potential for only negative or unfavorable interpretations of “placement prevention” as a 
solitary, distal outcome measure, we have developed a strategy to assess a variety of 
family functioning indicators along with the mandated statistic. Examples include; 

• service utilization subsequent to family preservation services; 



• general physical and mental health of family members; 

• school attendance and performance of school-age children; 

• involvement of children with the juvenile court; and, 

• subsequent involvement with DSS. 

These measures and others will provide a much more useful picture of the life- 
course trajectory of families following FPS than would be the case if we relied solely on 
whether the family is still intaa. 

As an aside, using “service utilization subsequent to receiving FPS” as an outcome 
measure may be argued to be either a positive or negative indicator. Some would argue 
that any need for services subsequent to receiving family preservation services is a sign of 
lack of effectiveness of the FPS intervention. Others would argue that if the FPS- 
receiving family is still intaa, then subsequent use of other services is no worse or 
different than another non-FPS family needing some services to alleviate family stress. 

Still others would argue that if the post-FPS services are initiated by the family itself, this 
is an indicator of family strength, not family weakness or deterioration. Whatever your 
position on the use of services subsequent to receiving FPS, Theiman, Fuqua and Linnan 
(1990) found that most families that initiate services af 'T’S do so within 30 days of 
FPS service termination, but many initiated post-FPS s jrvices at later points in time. 

While the meaning of this finding is subject to interpretation, the finding itself supports the 
contention that periodic, distal measures be taken in order to assess not only the durability 
of an intervention (as intended at the end of service), but also to detea other indicators of 
child or family functioning that may be related to the intervention. 

Another measurement issue of no small importance is that of linking outcome 
measures with the intended users of the information. Several recent evaluations of family 
preservation services (Rossi, 1991; Feldman, 1991; Yuan, 1990) have presented equivocal 
findings about the effectiveness of family preservation services. While the authors provide 
a variety of explanations for the lack of definitive results, I believe that these equivocal 
results are due, at least in part, to the resistance encountered when imposing an 
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experimental research strategy onto a practice environment in such a way that the people 
providing the data do net receive much benefit for their efforts. If there is little or no 
direct benefit to those providing the data, there is little incentive for them to take the 
research effort seriously. This problem is compounded when a measurement system spans 
more than one service environment. 

Practitioners and program directors in different child serving systems have very 
different philosophical and theoretical underpinnings for their programs. There are also 
differing political and fiscal pressures that must be accommodated in some way. Dore 
(1991) has suggested that one approach to designing outcome measures is to make sure 
that the outcome measures are tied to program goals of the relevant service systems, to 
differing fundamental theories of family dysfunction, and also to differing referral sources. 
Dore (1991) states that outcome measures will vary according to the child serving stream 
that guides [in this case] the FPS program; that programs in mental health streams must be 
evaluated according to the family’s role in caring for a special needs child; that for 
programs in juvenile justice streams you must include various measures of recidivism and 
community safety; and that in child welfare, outcomes must include child safety or well 
being, and family’s or caretaker’s ability to protect, supervise, and provide basic 
necessities (p. 134). The implications for measurement of outcomes is that systems will be 
more inclined to support data collection efforts if that system’s needs for information are 
represented in the outcome measures employed. 

Engaging caseworkers at an individual level is also important. If workers receive 
feedback on the outcome status of individual families or children, or if they can be 
provided with quarterly or even annual reports of family progress, they will be more 
inclined to willingly contribute data to the measurement system. This capacity will go a 
long way towards improving the reliability and validity of the measures. Jones (1991) has 
suggested that obtaining good outcome data requires multiple sources of data. He 
suggests families, caseworkers and collateral agencies (such as CPS, mental health, 
schools, and day care providers) as primary sources of data (p. 179). 



Integrating the ideas of several of these researchers, it seems logical to suggest that 
outcome measures should be developed in collaboration with caseworkers and supervisors 
in addition to the research a policy representatives normally associated with this task. 
Further, it is suggested that “management reports” and “practice reports” that rely on 
outcome measures be designed and provided routinely to those contributing to the 
measurement process as a benefit to them for their efforts. 

In closing, let me put forth three summative statements to help guide the activities 
for the remainder of this Roundtable: 

• An outcome measurement system must include child- and family-centered outcome 
measures as well as program-centered outcome measures. If only one category is 
represented in the measurement system the data are easily manipulated or 
misinterpreted; 

• Outcome measures must be relevant to the service environments in which they are 
applied, because different values are placed on different outcomes in the different 
systems; 

• Multiple outcome measures should be taken at different points in time begirming at 
intake, throughout intervention, at case closure, and periodically thereafter to 
assess not only immediate changes attributable to the intervention, but to test the 
durability of the intervention and assess unexpected or unintended consequences of 
the intervention. 

This is a tall order, but there are several new federal initiatives under way, such as 
AFCARS and SACWIS, that will improve our information systems capabilities. We must 
rise to the challenge of demanding that these new systems tell us what we want and need 
to know about program effectiveness, not just process. The results of this Roundtable and 
fiiture Roundtables can be used to inform the system development efforts about 
appropriate outcome measures in child welfare, to resolve some of the measurement issues 
relating to outcome measurement, and to promote the use of outcome measures to both 
improve practice and reform policy. 
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Second Annual Outcome Measures Roundtable 
Closing Comments by 
Patricia Schene, Ph.D. 



It is my responsibility to provide some closing comments summarizing the Second Annual 
Roundtable on Outcome Measures. I thought it would be useful to revisit the stated purpose or 
focus of this Roundtable, to summarize some of the main points raised by many of you in the 
course of our deliberations, and to surface some ideas or issues for consideration for the Third 
Annual Roundtable which will be held in San Antonio in the spring of 1995. 

The overall goal of the Second Roundtable was to have all of us contribute to a menu of outcomes 
for children and &mihes to use to guide the necessarily separate processes in a variety of 
jurisdictions to conunit to defining outcomes for child welfare services. 

The product of this Roundtable was not so much a list of outcomes and their measures but rather a 
greater readiness to undertake the process in individual states and counties as well as some tools or 
guidance from the experience of others. 

The question still remains as to how we can balance the necessary separateness of each local effort 
to choose outcomes while simultaneously recognizing the need for some agreed upon set of national 
outcome goals for the children and families who may reach the public child welfrure system. 

We also continue to stmggle— not as much as last year— but still we continue to stmggle with 
balancing the outcomes for children and Emilies with the more specific outcomes for pubhc child 
welfru’e. 

It may be helpful to review some of the issues and comments raised at this Roundtable in order to 
not only summarize our experioice but also to reveal the scope of issues that need to be addressed 
as we address outcomes and their measures. 



Issues and Comments 

One of the first issues raised can best be stated by the comment of Homer Kern and David Sheets 
that deciding on what are the “right” outcomes is not the relevant question. Rather we need to 
focus on what is the process through which we become committed to outcomes. June Cairns 
reminded us that we should not leave out private, commimity agencies in that process. Carols Sosa 
pointed out that there is no “holy grail” of outcomes. 

Another major issue raised is that we cannot continually alienate the public by refusing to accept 
some responsibility for basic outcomes such as safe, adequately functioning children. We need to 
relate our outcomes to public expectations. John Mattingly advised us to keep outcomes simple- 
simple enough for the public and the legislators to understand. If this is done well, we can use 
outcomes to get support for the work of child welfare. 

Dana Fabella pointed out that defining and measuring outcomes is not just a child welfare problem 
but a community problem. “I’m a piece of the system, not the whole system. We need to work 
with the community to share responsibility for outcomes.” 
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Ray Kirk reminded us of the broad relevance of our endeavor to define and measure outcomes by 
referring to the USA Today article on the impact of family preservation services. Another example 
he gave was of the North Carolina legislative session on crime dealing with prevention where the 
clear hope was that family preservation centers would produce outcomes in terms of the prevention 
of crime. 

Another major issue raised w'as that we continue to be concerned about the political risks of 
determining outcomes and then failing to meet them. Bill Hausman reminded us that we are 
currently failing miserably to meet the pubhc’s expectations and that itself should alleviate the 
political risks of actually defining and measuring outcomes. 

Sylvia Pizzini pointed out the importance of private agencies “stepping to the plate” in helping to 
achieve outcomes. June Cairns agreed but added that it was also necessary to include private 
agencies in the plamiing processes where outcomes are established. Others pointed out that the 
planning process soon to be underway for the Family Preservation and Family Support Act will 
provide some forums as well as incentives for private agency and commimity involvement. 

Another issue raised related to the operuiess of the current system to changes that outcome 
measurement might direct. How can we ourselves get out of the traditional social work models we 
created for intervention and be willing to deliver services differently if outcomes can be more 
effectively. 

Comments were made again this year about the very real issue of resource limits. When we all 
come to the table with our marbles, there are still not enough there to tmly change the lives of 
children and famihes. Yet Carlos Sosa suggested that instead of just saying we need more dollars, 
we need to say that we have a lot of money being spent now and the important fijcus should be on 
what V e can commit to achieve with those resources. Richard Cahca said we need to realize that 
it’s not just a matter of resources, but that we have to recognize that the technology' may not exist 
to achieve outcomes. Some of the problems of children and families are intractable even with 
dollars and time. 

Another important issue raised was the need to collect data on outcomes. Will Johnson pointed out 
that we know so little that is based on empirical data. For example, his research reveals that only 
23% of opened cased of child abuse and neglect have a recurrence of reported maltreatment over 
the following two years. That could be seen as a 77% success rate. The child functioning 
workgroup agreed that we need to have good data so we can be held appropriately accoxmtable. 
On the other hand, we must realize that we are held accoimtable whether we have good data or not. 

A point was made about the relationship of the authoritative or involimtary nature of child welfare 
and the definition of outcomes. Who gets targeted for authoritative intervention vs. voluntary 
intervention is very relevant as we try to define and measure outcomes. Sylvia Pizzini pointed out 
that we need to make a distinction between what outcomes we would like vs. what outcomes we 
demand when families are involved involuntarily. The need to articulate outcomes through the 
juvenile and l^mily court as well as through the child welfare agency is especially relevant for this 
population. 

The distinction between outcomes and process was raised by many during the course of the 
Roundtable discussions. As Russell Cardamone pointed out, we need to be open to using the 
process piece to help realize outcomes. Outcomes do not replace process, but go beyond process. 







Another way of looking at this issue is to realize that process is nested in outcomes, rather than 
thinking that a process orientation is the opposite of an outcome orientation. 

The challenge of implementation was raised by many. It is one thing to define and measure 
outcomes and another to have that information be used in meaningful ways. Eric Bjorklund said 
we need to be open to measurement of residential care fecilities on two key outcomes; movement 
of children to less restrictive settings and school performance. The expectation would be that the 
tracking of performance on these measures will affect program emphasis. The example given often 
is the impact on the airlines of tracking on-time arrivals. 

In implementation it is important to clarify values before we can move to outcomes at the 
community level as well as the administrative level. As Lloyd Malone stated, implementation is 
affected by what agencies are participating and by what the commxuiity can support. 

Another issue raised was the tension between the need to evaluate what we are doing now and the 
need to develop outcome based measures, as David Bernstein said. Madeline Kimmich suggested 
that we have to operate at both levels simultaneously. 

The role of leadership is crucial to the process of developing outcomes and their measures. We 
caiuiot measure outcomes unless we are very clear what our program goals are. Political 
constituencies will always be with us and good programs may not survive without pohtical 
support. But, outcome measurement is a professional issue as well. We must hold ourselves 
accoimtable for the important work we do. This requires leadership and sustained commitment to 
the development and use of outcome measures. 

Defining Topics for Future Roundtables 



It was suggested that we address the incorporation of outcome measures into the child welfere 
reform agenda in terms of cross-system collaborations, earlier intervention and prevention 
programming, and building neighborhood or community systems of care. 

Another topic suggested is the assignment of key accountability for outcome achievement; if it is 
everyone’s job, it may become nobody’s job. How can responsibility for outcomes be shared while 
simultaneously estabUshing accountability? 

The tools for measuring outcomes also need to be addressed. 

The issue of federal monitoring such as 427 reviews was raised in terms of incorporating or 
substituting outcomes for existing process in federal accountability as well as in state requirements. 

More attention needs to be paid to the specifics of using process measures currently required in the 
delivery of child welfare services in a way that they can be viewed as nesting within an outcome 
based system. 

Implementation and utilization of outcome measures is a vital topic where experience needs to be 
shared. 
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We need to address the linking of interventions to outcomes. This is a research as well as a 
practice issue. What interventions are most successful in producing what outcomes for what 
clients? 

We need more carefully drawn distinctions between outcomes a'-., their indicators. 

More attention would be welcome in how to go about fecilitation of the process of community 
based participation in the definition of outcomes and their measures. 



Conclusion 

This Second Roundtable was considered very productive. The preparatory documentation 
provided gave participants a framework within which to work. The definition of four broad 
outcome areas— child safety, family continuity or family preservation, femily functioning, and child 
functioning were considered helpful outcome categories. The workgroups made a great deal of 
progress in developing indicators for each of these outcome areas. Moreover, the presentations of 
experience and insights were very helpful. The Roundtable process is meeting its goal of providing 
an annual fonun lor the field to share progress of outcome measures and to work together to define 
a menu of outcomes and their measures that will foster developments in many different states and 
communities. 
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DAY ONE: THURSDAY, APRIL 7, 1994 



1 0:00 - 1 :00 Registration: Travis Room 



1:00-1:15 


Weir me and Introductions 


Speakers: 


Kittle Arnold, Arapahoe County Department of Social Services 
Elizabeth Thieiman, National Association of Public Child Welfare 
Administrators, APWA 
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1:15 - 2:00 


The Importance of Measuring Outcomes in Public Child Welfare 
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John Mattingly, Senior Associate, Annie E. Casey Foundation 


2:00 - 2:30 


Review of Purpose and Agenda for the Second Roundtable 

• Summary of Issuesf Emerging from the First Roundtable 

• Overview of the Agenda 


Speakers: 


Patntia Schene, American Humane Association 
Dennis Orthner, University of North Carolina, Chapel Hill 


2.30 - 3:00 


Discussion: The Value and Challenges of Developing and Measuring 
Outcomes in Child Welfare Services 


Moderator: 


Carlos Sosa, Los Angeles County Department of Children's Services 
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3:00 - 3:30 


Break 


3:30 - 4:30 


Review of Background Paper: Defining Outcomes for Child Welfare 
Services 

• Framework 

• Target Outcomes 


Speakers: 


Dennis Orthner, University of North Carolina, Chapel Hill 
Nancy McDaniel, American Humane Association 


4:30 


Closing Announcements 



6:00 - 9:00 Evening Reception: Jefferson Manor 



ALL GENERAL SESSIONS WILL BE HELD IN THE TRAVIS ROOM 



DAY TWO: FRIDAY, APRIL 8, 1994 



8:30 - 12:00 Presentations of Experience 

• Process or challenges in defining outcomes 

• Specification of outcomes/indicators 

• Implications of outcome definition for child welfare system 

• Results /findings to date 

• Future direction 



Moderator: Nancy McDaniel, American Humane Association 



Colorado Ann Anderson, Colorado Department of Social Services 

Kittie Arnold, Arapahoe County Department of Social Services 
Terri Bailey, Piton Foijndation 

Lloyd Malone, El Paso County Department of Social Services 



Texas 



David Sheets, Texas Department of Protective and Regulatory Services 
Homer Kern, Texas Department of Protective and Regulatory Services 



10:00- 10:30 Break 

Calif Will Johnson, Alameda County Social Services 

Utah Earl Bassett, Utah Division of Family Services 

Eric Bjorkiund, Utah Youth Village 



1 2:00 - 1:30 Lunch - On Your Own 



1 :30 - 1 :45 Definition of Work Group Tasks 



1 :45 -3:15 Work Groups Meet Defined around Target Outcomes 



Breakout Rooms • 

Lafitte 
Laredo 
Midland 

Corpus Christi • 



Clarify definition of each target outcome; child 
safety, child functioning, family functioning, family 
continuity/preservation 

Review illustrative examples of outcomes 



• Generate list of additional outcomes 



3:15 - 3:45 


Break 


3:45 - 5:00 


Reports from Work Groups 




• Report back to Participants 




• Reaction and Discussion 



Facilitators: 



Pat Devin, Texas Department of Protective and Regulatory Services 
Carlos Sosa, Los Angeles County Department of Children's Services 



DAY THREE: SATURDAY, APRIL 9, 1994 



8:30 - 9:15 


Work Group Session: Prioritization of Desired Outcomes 


Breakout Rooms 
Midland 
Corpus Christi 
Travis Room 


• Prioritize based on input from previous day's discussion 

• Identify no more than five outcome indicators for each target 
category 


9:15 - 10:15 


Report from Work Groups and Participant Reaction 



Facilators: Pat Devin, Texas Department of Protective & Regulatory Services 

Carios Sosa, Los Angeles County Department of Children's Services 



10:15 - 10:45 


Break 


10:45 - 11:45 


Getting Started 



Speakers: Dennis Orthner, University of North Carolina, Chapel Hill 

Ray Kirk, University of North Carolina, Chapel Hill 



• 


What have we learned about measuring outcomes? 


• 


What current vehicles or initiatives support development and 
implementation? Who do you involve? 


• 


What strategies encourage success? 


11:45 - 12:00 


Summary of the Roundtable 


Speaker: 


Patricia Schene, American Humane Association 


12:00-12:30 


Closing Announcements and Planning for the Third Annual Roundtable 



Moderator: Bizabeth Thieiman, National Association of Public Child Welfare 

Administrators, APWA 



12:30 



Meeting Adjourned 
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